Browsing my B&M OED I see that the original definition of enormity was something along the lines of 'divergence from the norm.' Enormous had a similar meaning. This goes back to the late 15th century. The limitation to divergence from the norm in size, and then only in the sense of excessive size, dates to the late 18th century. Enormous took on the sense of 'great size' in the mid 16th.