limit is an infinitive and, as Jackie stated, doesn't need the "to".
But what do you make with a sentence like: We wouldn't have needed to limit ourselves.. ? "limit" is infinitive here as well, or isn't it?