Context is everything.
Here’s the thing…it varies by region and local custom. If you call the person by the label they wish to be identified by, and they don’t take offense to it, one’s common sense should be able to distinguish between a racist remark and a lack of sensitivity…in many areas of the American South, some of the older generations (blacks and whites) don’t take offense to the term “colored” as the label was a practical one in their previous way of life (a society that once accepted de facto segregation). There are clearly other terms used in the South that are patently offensive, which in other regions we wouldn’t necessarily identify…for instance, if a person called another person “boy.” That clearly harkens back to antebellum references and ways of thinking and judgment.
If the Chinese colleague is accustomed to and not offended by the term “colored”, then unless your boss was using it in an expressly pejorative context, it probably is of little concern. “Get that colored girl to introduce you to the group” or “that colored woman will clean up our mess when we leave for the day” would be offensive remarks in context, implying that her ethnicity determines her class / position, even though that clearly would not be the case. Also, those comments could imply that her sex would classify her in a subservient / subordinate role…creating or reinforcing imaginaries about the environment would be the danger in such comments.
This may be why when personnel are transferred or hired from outside the region, their “regionalisms” not only include cultural differences, but even language and vernacular.
Also, generational gaps may play a role in this as well.
Now it probably would have been a lot clearer delineation if your boss would have referred to her as a “Chinaman” or some other denigrating term, but perhaps your question has broader implications (calling all non-white/non-caucasian people “colored” or “of color” has the same connotation of generalizing white and non-white).
In a recent journey to a family reunion in Panama, I witnessed some of my relatives referring to the dark-skinned (African descended) Panamenians as “negro” and I had to take a step back and realize that that was a very different usage than what we have come to know in American history and culture (“Negro” being an archaic term compared to the current usage of the label “African American” in America, but in Latin America, a simple description of skin color-“Black” in Spanish is “Negro”).
Furthermore, one friend of the family, who was light-skinned, was reffered to as Guero (pronounced “who-ed-O), which translated means “white guy”...over there no one took offense to it, but here in America, it could be misconstrued as racist.
Finally, during my college years, I attended an art college that had a majority population of international students; it was clear to me that the Africans from Africa did not like to be called “black” or “African American” and that they preferred “African.” Also, some Asians did not want to be referred to as Asian, like some Koreans I was acquainted with, wanted to distinguish themselves from other East Asian ethnicities (try calling a Korean by something other than Korean-i.e. Japanese or Chinese!)...just my take on it.