Need is a strange little modal verb. "We need not go", yet "we don't need to go." Other modals (could, would, should, etc) are much more clear-cut. That's not much of an explanation, just an observation.