I think this line is unutterably sad: I doubted if I should ever come back.

Not sad, surely, but realistic. To me, this poem underlines the futility of wishing for what cannot be achieved, and to accept that one is continually making choices and that one is continually having to accept the consequences. In some ways, this is as fatalistic as Omar Khayyam - perhaps that is why an Autumn, rather than a Spring setting is used? Autumn is the time of fruit which will become the next stage of development, just as your choice of road leads to the next fork where you must make a decision and lead on to the next stage of your life.