I think it crosses the line when the difficulty comes not from having to use applied strength or skill, but from having to deal an element that only serves to be frustrating and hard simply for the sake of being hard.
A "Challenge" is defeating a boss who is powerful and has a complicated attack pattern. "Difficulty" comes from having to do it without taking a single hit, and if you die you have go through the entire level again.
--
Okay, I rolled a 14. What's that mean? Hsu
That you're a cheater. This is a 12-sided die. Chan