when I was in little league, I was banished to *right field -- most little leaguers are right-handed and right-handed batters (all else being equal, as it generally is in little league) usually hit to the left side of the field.

here is what I found the last time this came up:
William Safire poses two theories (in his book "I Stand Corrected"): 1) it was an insult hurled at kids stupid enough to buy left field seats at Yankee Stadium in the days when Babe Ruth (great hitter, poor fielder) patrolled right field for the Yanks [but weren't most of his homeruns hit to left?] 2) at the old West Side Park in Chicago there was a mental hospital in back of left field.