the difference is between this phrase and the less-PC phrase " c o l o u r e d people."
It's because the term colored people was used at a time when they were the target of heavy, open prejudice--a time that many people don't want the memories of being brought up again and again.