I'm with Jackie, here. It'd be rough to pin down a hard and fast rule with "little" and "small". Some of the rules are likely to vary from place to place. Jackie's example of "little brother" is a case in point. I've heard instances of diminutive women referring to some monstrous hulk as "my little brother." On the other hand the term "small minds" is belittling.