at first blush you'd think it would be the coldest, having gotten gradually colder as the days shorten
Um, well--the number of daylight hours isn't necessarily a reliable indicator of temperatures. Depends on where on earth you're talking about, I reckon.
I have always thought of the seasons starting when the calendar says they do: starting into their appropriate weather, which then (usually) peaks somewhere near the middle of the three months, and gradually tapers off towards the end of that period.