The three citations at OneLook all document p's and q's as a plural noun with the apostrophes. This is from American Heritage:

p's and q's

PLURAL NOUN: 1. Socially correct behavior; manners. 2. The way one acts; conduct: was told to watch his p's and q's.


Here's a discussion of the possible origins of the term from Word Detective:

http://www.word-detective.com/052699.html#psandqs