From James Brigg's "Origin of English Sayings", I got this suggested explanation:
Fig: I don't give fig indicates complete lack of concern about an event. It comes from the Spanish Fico (= Fig) which gave its name to a traditional gesture of contempt made by placing the thumb between the first and second fingers. The gesture was common in Shakespeare's time and was known as The Fig of Spain. The modern-day equivalent is the "V"sign.


noho ora mai
http://maxqnzs.com/References.html