I've never attempted to make sense of it from a representational graphics aspect, but have always known the Xs to be the kisses and the Os to be the hugs. My own recollection is that the "Os" are fairly recent. This impression seems to be supported by the fact that Webster's Dictionary includes an entry for this use of X, but nothing similar for O.

From Webster's:

X, symbol. ... 2. (used at the end of letters, telegrams, etc., to indicate a kiss.)