gender-neutral term ... as in "a gentleman's game"?

Not really needed is it? A gentleman's game - to me - a game only gentlemen (read rich) men can afford to indulge in. Like Polo or maybe Horse Racing (where you own the horse!)

Let's see would a woman's game be -- having baby? or if that's too heavy -- synchronized balletic swimming.