Physical attraction isn't really something you consciously decide upon. I would think that in this case it would be because most black people have certain physical characteristics that you, on subconscious level, don't find attractive.