surely 'oxbridge' itself must annoy one or the other, if not both...

Consider the term 'Oxbridge' as the equivalent of 'Ivy League'.
Either can be put-down or praise, depending on context and/or tone of voice.

Which leads to a question - the derivation of Oxbridge is pretty obvious - where did the term Ivy League come from?