Thanks all! As for me...
aroma - a smell that can be tasted. Definite association with food.
As for the other two...this is not spontaneous, I had to think about it and this is what I feel
fragrance - is diffuse, a smell that tends to disperse into the air easily
perfume - is a concentrated, localised smell. Civet is a perfect example, I think (never smelt it myself, but it has always *sounded potent and concentrated).
To elaborate: we'd say the fragrance of a garden full of flowers. We wouldn't say perfume, would we?

What is the etymology? Is there something there? And TEd, I laughed, but am sure all the ladies will agree that all three can be pretty upmarket.