Boy howdy! Now I'm all self conscious about it and I don't know if I pronounce the L or not. I kinda think I'm with tsuwm here and sort of almost maybe pronounce it. It's probably one of those words like salmon where the L was added back in when we discovered it was from the Latin, which did have the L. The L was lost in Old French before we got the word salmon. We called the fish leax before the French word came in and muscled that word out.

A quick check of my Random House Dictionary shows that the L was there in OFr when we got almond from them so I guess there's no reason not to pronounce it.