The N64 genre-wise is probably most known for four things (in my opinion at least): 3D platformers, adventure games, FPSs, and racing games. The stock rushmore does cover all of these and Mario Kart and Goldeneye also cover the 4-player stuff. I would agree the N64 basically needs a 4-player game and preferably more than one.
Though, another part of it, is that these four particular games are absolutely gargantuan compared to the rest of the library. To elaborate on MK64 (probably the most contentious of the big four going up) a bit, though obviously SMK is where the series started, most of what we (and even the general public) associate with Mario Kart now - hectic 4-player multiplayer, the 16-course standard (and a lot of tropes associated with the courses - a Luigi course near the start, a DK jungle track, a single Bowser's Castle near the end, etc.), the standard 8 character roster, more of the standard items, and perhaps most importantly the more ambitious track design - comes from MK64. It is definitely the Melee, Sonic 2 or Mega Man 2 to SMK's Smash 64, Sonic 1 or MM1. There is a reason the "Mario Kart clone" specifically cropped up after MK64, not SMK.
Born to lose, live to win!