I had always assumed that it was an extension of the phrase, "I'd give my right arm to ..." when someone desires something so strongly that they would make massive sacrifices for it to happen. (to be ambidextrous, for instance )

But I must say that dxb's explanation looks to be more academically sound.