it just means on average white people have less hardships (usually caused by race) than minorities.
No. It means that there are specific hardships that the class of white people don't have to face, meaning that, on average, it's difficult to understand how it feels to be . One such example would be how black people get pulled over for being black. For that reason, whenever there's a police car on the road, a completely innocent black person might feel exposed/threatened/nervous in a way that is difficult or rare for a white person to experience.
In these contexts, a privileged group refers to a group that is exempt from specific hardships endured by an oppressed group. It's not "on average white people have it better". It's "even if a white person has it much much worse in any other sense, they don't have this specific problem".
I've been pulled over for being white. So much for white privilege.
Also, if you think white people can't understand what it's like being a minority, go live in a ghetto for a year as the only white family in a five mile radius.
That's my childhood. ---
'It's okay that those gangbangers stole all my personal belongings and cash at gunpoint, cuz they're building a rec center!' - OneTimeBen