As an insult, I'd invariably think it male. In the phrase 'bastard child", I think of it as gender neutral.