I've always used "the short end of the stick"; in fact, I used it today somewhere. I guess that's a nicefied "shit end of the stick".

Yup, Bean..."short end of the stick" was the common usage in polite company.