The phrase might be connected to the Army-Navy football rivalry. The goat has been the Navy mascot since at least 1893 (see http://www.nadn.navy.mil/PAO/facts/Goat.html) and I believe that the two academies have tried to kidnap each other’s mascots (Army’s is a mule) ever since. The date and place of first usage would fit, and the meaning of “getting one’s goat” to be an irritant would be a direct lift.

This was actually the first thing I thought of, but I figured it would be too recent a fad to have created the phrase, but it looks like at least a possibility. Good thinking (and research)!