Quinion has a good run-down on the word octothorpe. As far as ampersand goes, it's a convention from old glossary entries that read "& per se and", i.e., "The sign &, by itself, means and.". You can refer to single letters as, A per se, in the OED2, "the letter A when standing by itself, especially when making a word", &c. (Reminds me of the Dord entry in the Webster's 2nd.)

Ceci n'est pas un seing.