My guess is that the phrase arose before the invention of envelopes, when it was routine to fold up a piece of paper and write the address on the outside.



TEd