Welcome, JeffK.

If you google the term with and without the L, you'll find both are pretty common: I agree with you, though, that buff does sound funny. I'm sure one of our resident scholars will explain the metathesis or whatever is going on here linguistically.