The insult must start in the mind (however small) of the speaker. A neutral term becomes offensive when it is used as an insult ofen enough to absorb a connotation of "I use this term because it describes someone whom I consider less than me". If "plaid" was coined as a perfectly friendly term for my mix of Irish and Scottish and then later used as an insult it would soon be one.
Personally, I think that the statement "He looked drunk." would have been sufficient without a racial qualifier.