Can we really make too much of a word’s (auxiliary verb, noun, whatever) literal meaning?

In this case, yes. It's performing a function that transcends its literal meaning. What does the literal meaning have to do with any of those various future auxiliary verbs I mentioned that were used in Vulgar Latin in place of the conjugated form of the verb in Classical Latin? Perhaps at some time there was some rationale that went on in the minds of the speakers of Vulgar Latin. Whatever it was is lost and we are left with the remnants. We understand I have to as a modal form of the verb expressing compulsion just as we understand I'm going to as a future. Inquire, if you will, into the origins, but don't bother trying to parse it according to the present day literal meaning of the words involved.