No matter how innocuous a term is originally, if used to describe a minority, it will eventually sound like an insult because of the ignorami who use it as an insult. Coloured people was at one time the new and non-stigmatising term.
I do have a problem with the term "people of colour" because it forgets all of the other colours out there. (and because it makes me feel somehow grey and colourless, like a poor quality black and white photo.)