According to Why You Say It, by Webb Garrison, stable attendants used to sooth a racehorse by stabling a goat with him as a companion, and, once the horse became attached to the goat, it created an equine crisis to remove the goat, and that is why a person who makes you frustrated or angry has gotten your goat. Garrison cautions that this explanation is based on oral tradition.