A "colored person" is a black person. That's the usage that's long established, and trying to change it would be offensive and difficult.
"Person of color" immediately prompted "person of pallor", which immediately nailed the difference. There are whites, one group; and then there's everybody else. The term was coined when the commonality wasn't all that widely felt, when there wasn't a large non-white cohesive bloc. (There still isn't, but some think there is.)
Jorge, Ibrahim, Ling Yao, Ashok can each be a person of color but aren't "colored people". Formally they're not that different, but functionally they're worlds apart.
At the same time we've disposed of "races", which means that Jorge and Ibrahim, probably considered "Caucasian" 50 years ago by phys anthropology folk, now aren't Caucasian because the category is asserted to not exist. For those for whom "white" = "Caucasian," like the census, they are, however, white; for those who like "person of color" they are forever non-white. Those caught in the middle are those for whom "white" is a shifting category--just as Irish didn't use to be white but now are, many Latinos and Middle Easterners that weren't "white" 30 years ago are now lumped in with "white" by many. That neuters the strict, unyielding dichotomy that many activists need to create unity.
Still, it's a fairly innocent error, one you can get to several ways ("person of color" --> "colored person" or just "black person" --> "colored person," if you can't lexically access "black" before "colored" pops up and passes muster.) Many of those most prone to wearing "Don't Judge" t-shirts among people I know are, not all that strangely, the harshest judges.
I wonder if the woman was primed by having heard "colored people" recently.
= new reply since forum marked as read