Verrry interesting, as there are all sorts of potential connections here.
The closest equivalent to losing your rag is simply losing your temper, and my first thought was that the phrase may be a corruption of "loosing your rage". However, it feels [very scientific again, Fisk] more closely related to "ragging" as in winding up or taking the piss, i.e. someone rags you and you lose your rag (which is implicitly not the reaction of a good sport or gentleman).
As to ragging=winding up, that's probably related to teasing/playing with animals like dogs (especially) and bulls (like a red rag to a bull).
The more I think about it the more the dog connection seems appropriate. If you manage, by some miracle, to pull a rag out of a dog's mouth, it'll go for you like a mad 'un. And a big dog that's not been raised as a family animal will be genuinely dangerous (and angry) if teased past a certain point.
usual disclaimers apply
