Some people say profane things when upset. Just saying the N-word isn't indicative of racism. Unless he starts casually referring to black people as N-words, I'll file this under no big deal.
I remember I worked with some rednecks a few years ago. They referred to everyone as N-words. Hard R. Each other, other white guys, hispanics, black people, asians, etc. Just everyone. It was a bit disconcerting at first to hear it said so liberally and out of the context you'd expect. Granted, some of them were explicitly racist.