He does have a point. Depending on the generation and time you grew up in, you may have had more exposure to the environment in which people commonly said colored, negro, or even blackie as these terms were common in the mid-1900s. As someone who was a child in the 90s and grew up in the 2000s, these terms are completely insensitive and offensive, but thats just because I was raised in an environment that disapproved of these terms.
For someone who grew up in the 20s, 30s, 40s, or even 50s, these terms were very common and the environment was welcoming to them, despite their possible negative connotations.