...and then was changed in some movie to gone south. Possibly Tommy Lee Jones in U.S. Marshals.

oh it's older than that, although I don't know how much:
1977 Time 12 Sept. 46/3 Some striped jobs look like pousse-café or rugby sweaters gone south.

edit: maybe it's from this Warner Brother's cartoon: Dog Gone South (1950) with Colonel Shuffle and Belvedere