Agreed. Another way of seeing the distinction is that those who swear would swear at someone; you would never swear "with" or swear "to" them.