just knowing it was the longest day made it seem more special

It may have been the longest day but it wasn't the latest sunset. That would have been sometime later. See http://geography.miningco.com/science/geography/library/weekly/aa061500a.htm.
You can retrospectively revel in the fact that you had up to a week of evenings when you could stay out past the time you could have on the evening of the solstice, if only by a minute.