It is, so far as I know, a "townee" word (i.e., originating among urban rather than rural dwellers.)

The birds found in towns are, generally speaking, of the smaller varieties - sparrow, starling, thrush, blackbird, etc. They, naturally enough, have small heads so it would be assumed by "townees" who know no better, that the capacity, therefore the intelligence, of the brain is small.

Hence the unjustified pejorative effect.