'downhill' borders on being another of those enantiodromic words; as you imply, "it's all downhill" could mean it's going to be easy or we've reached a high point and it's just going to get worse.

http://members.aol.com/tsuwm/