2006 killed it for me. As bad as Summer 2004 and the villains contest were, I still enjoyed Summer 2005. 2006 marked the beginning of contest gimmicks, which not only caused its own bracket to be crammed full of complete fodder (thanks to the asinine male/female idea) but served as an admission that these contests had run their course. Four-ways were an interesting idea, but I just plain wasn't interested by then, since the so-called "magic" was gone. Plus, Summer 2005 would have served perfectly well as a sendoff to the idea.
How to bring the magic back: Game contest. 1v1 polls. Bracket seeded normally and not organized by year, generation, system, or anything like that. Throw FF7 and the N64 Zelda games into a Tournament of Champions. If this doesn't get decent vote totals, we can all admit that contests are truly dead.