I've always suspected that my family swung the 'other' way

Oooo shanks. You don't want to say that here. Here, if your family had "swung the other way" you would not have made any babies at all. An important phrase to avoid if you ever come on a visit and you don't want to lead any gentlemen on.