Brian sez:
If the "ham" means "town" like I suspect, it probably is a derogatory derived from a placename.

Hmmm... does the term "village idiot" come to mind?