I wouldn't belong to a club that had someone like me as a member... -Milton Berle or Groucho Marx?

Groucho. Definitely Groucho. With a word-order twist:
"Any club that would have me as a member I wouldn't wanna join! [making-with-the-eyebrows]"

But I forget which movie.

PS. Is there a name for that gesture of raising-and-lowering-the-eyebrow(s)-two-or-three-times archly ?