I linked to the Carly Simon version (above) because it includes the following intro which relates directly to this discussion (and which Frank skipped):

When the sun is high in the afternoon sky
You can always find something to do
But from dusk till dawn as the clock ticks on
Something happens to you..

In the wee small hours of the morning
etc.