Hoi polloi is Greek for "the common people." It is often incorrectly taken to mean "high society", probably because of the way it sounds. Here's a usage note from Atomica*:

USAGE NOTE Hoi polloi is a borrowing of the Greek phrase hoi polloi, consisting of hoi, meaning “the” and used before a plural, and polloi, the plural of polus, “many.” In Greek hoi polloi had a special sense, “the greater number, the people, the commonalty, the masses.” This phrase has generally expressed this meaning in English since its first recorded instance, in an 1837 work by James Fenimore Cooper. Hoi polloi is sometimes incorrectly used to mean “the elite,” possibly because it is reminiscent of high and mighty or because it sounds like hoity-toity.

*Atomica is a handy little reference tool which you can download. If interested, click on http://www.atomica.com