"That definition seems a little harsh. I'm sure that my aunts would have said things like, "she's got a fancy
man" meaning someone who dressed or behaved as if they occupied a higher station in life - a sort of
spiv, a man in a suit when few around would wear them"

I always understood "fancy man" to mean merely "lover".