per the available citations, the sense shift from being 'some thing that is hard to swallow' to 'a disagreeable person' seem to have occured fin de siecle in the Yale/Harvard environs. (the former figurative sense goes wayback to C16.)