It's absolutely fair to say that, at least historically, there have been barriers to non-whites attaining positions of power.
However, the VAST fucking majority of white people have a snowball's chance in hell of ever getting into those same positions of power either. We're just the same color as people who do. And that's why it's blatantly insulting and racist when people assume we're "better off" just because of it. No, white people aren't privileged; rich people from political / corporate dynasties are better off. Yes, most of them happen to be white; but that doesn't change shit about the fact that most white people are not from such backgrounds.
Privilege is so much more than ease of attaining positions of power.