The trouble is, I suppose, that abbrevs and initials were originally used by Victorian clerks to make life easier for copying letters (before the days of carbon paper or typewriters) They were never intended to be used in speech, and the word written "Xmas" was supposed (and still is!!) to be pronounced Christmas.

Goodness knows why we have carried clerkly expediency into verbal laziness. Maybe because we are lazy!!