Board 8 > An analysis on Guru bracket winners and amount of risks taken

Topic List
Page List: 1
ZeldaTPLink
03/08/20 3:25:32 PM
#1:


Greetings! This topic is a new project I've made that should be interesting to all of you stat nerds.

This idea came up about a week ago, when I saw people discussing the need to take risks and avoid making their bracket too close to the cookie/chalk, and "get away from the pack". WHile that is usually seem as common board knowledge, I wondered if it actually held true in practice or if it was just a cultural thing that didn't actually increase an user's chance of winning. Then I realized I could investigate the results of the Guru Contests, since those almost always come with the risks for each player calculated, and see if I could find some pattern that justifies, or denies, the pursue for risks among bracket makers.

In case you don't know, a "risk" is when you make a pick that goes against the board consensus. If most people take Link to beat Cloud, and you take Cloud, that's a risk. You can find out how many risks you took, during the contest, by checking the Guru page. Before the contest, you can do it by counting your risks in the Board Odds Project. Now let's move on.

Objective: to find out if there is a correlation between number of risks taken and bracket score, or chance of ending in 1st place. If there is, figure out the optimal number of risks to take to maximize one's Guru winning chances.

Hypotheses:

1) Following the cookie is good. People who follow the board consensus tend to score high compared to most of the board. People who make tons of risks tend to finish far away from the first place. The board knows their stuff, after all.

2) If you are close enough to the cookie, it doesn't matter how many risks you take, because each individual pick is not more or less likely to get matches right than the cookie pick.

3) However (and this point denies #2 to an extent), there is the matter of bracket eliminations. If your pick is different enough from the cookie, you get a little more room to make mistakes without being eliminated by another bracket who was too similar to yours but didn't make that specific mistake. If you are almost identical to the cookie, you might not get that room, as there will be multiple people to capitalize from your mistakes. In this sense, while the cookie is not more or less likely to be 100% right than any other random bracket, it is less likely to score more points if it gets a few things wrong.

To test those hypoteses, I put all the rankings from the 12 guru contests onto a spreadsheeet, as well as each players' respective scores, and number of risks taken. Special thanks to @Ngamer64 for providing me with data on Best Series Ever, which allowed me to calculate its risks myself since those weren't in the guru site.



The first two lines present the total number of picks each user had to make, and the maximum number of points someone could make in each contest. Winner Risks shows how many risks each guru winner made, and the following line calculates its % on Total Choices, in order to compare different-sized contests. Average Risks gives you the average of risks taken for all users. Finally, Winner/Average compares the winner's risk taking to the rest of the field.

Looking at Winner Risks (%), we can see most guru winners take a number of risks in the 10-20% range, which seems to be a trend. The average for all contests is 14%.

There are three major outliers: Years, with 3%, Rivals, with 6%, and Characters 2008, with 29%. The first two were gimmicky contests that probably explain the higher level of chalkiness. Personally, I was here for Years, and I remember there were very few debated matches at all, so there just weren't many viable risks to take. Rivals was probably a similar case.

2008 is a weirder case, with the winner taking a grand total of 36 risks. It gets wrider when you look at the ranking, and realize the 2nd place also took a huge number of risks (37), but from the 3rd place onwards, the gurus took risks closer to the global average, although if you keep going down, a lot of brackets above 20% can be seen. It's hard to say if there is some meaning to this or if it's some statistical anomaly. If I recall correctly, 2008 isn't exactly known as a wacky contest, since it ended with a Link win and had no major rallies. Its counterparts in the 4-ways format, Characters 07 and Games 09, had their winners take 18% and 13%, respectively, so I'm not sure if it's the format either (though Characters 13, which was 3-ways, had a rate of 20%, which suggests multi-way contests favor risk takers a little more than the average, if you ignore Games 09).

The lack of gurus going below 9% except for those two gimmicky mini-contests suggests there is an optimal floor to be pursed, which gives strength to the Hypothesis #3 I outlined in the beginning. But then I decided to compare the winner risks to the average guru risks and the results were... not very meaningful. It seems the ratio between the winner and the average risks is all over the place, and the average of the averages gave me 16%, which is not much different from the 14% of the winners. All of this suggests a lack of correlation between risks and bracket performance.

So I decided to dig deeper. I put all the guru picks on graphs, to see if there were any visible trends.

Note: the graphs below don't actually show every single bracket. Each point corresponds to 7 brackets, with their values averaged. This is to make the lines easier to see, but does hide some points, most notably the guru winners. I believe this level of smoothness did not compromise the results very much, as long as you use it along with the table above.



The values for both bracket score and risks are averaged to each contest's available points, and total picks.

First, we can see a common trend where each graph starts at a high point, then drops down. This confirms Hypothesis #1: people who take too many risks lose. But, after you reach the top part of the guru rankings, the trend becomes less pronounced. That usually seems to hapen below the 20% risk mark.

For Series 06, there is a large number of people with similar risk levels, then it goes down, but then the winner then spikes again to 19%.

For Chars 06, most of the field has risks below 10%, but when you get to the top 3, they go above that mark.

Chars 07 has a number of risks actually above 20% in most of the list, stopping at 18% in the 1st place. The 2nd place, btw, had a whopping 29% risks, so this came close to being a very different story. But in the end, the win went to the one who stopped below 20
... Copied to Clipboard!
ZeldaTPLink
03/08/20 3:25:57 PM
#2:






This second batch of graphs seem to show clearer slopes in general. Maybe the board got better at contests after so much time.

Games 09 is a very interesting case because the norm is huge risks all across the field, with a lot of people over 25% near the top score. Yet, the winner only took 13% risks. This means having a lot of risks helped you get points in 2009, but to actually win the Guru, you had to shoot close to the magical 14%. This does fall in line with the theory I made above that multi-ways favor more risks, though, even though the user who finished #1 avoided that trend himself.

The first Game of the Decade has a lower average score than most. Yet, the risks were very concentrated under 15%, with the winner risking 12%. A lot of people risked a lot less, actually, getting close to 5%. Comparing it to this year's contest, I give the impression that people all followed the pack in what looked like a more chalkier bracket, yet when the matches actually happened, a lot of unpredictable results happened, and the people who breached that 9% limit did not succeed.

Chars 10 also showed a similar risk trend to GotD10, although the average score was much higher.

Rivals 11 has the guys at the top all going below 10%, with the winner going below 6%, which suggests a really chalky bracket where the chalk actually succeeded.



This batch of graphs also seems to have an obvious downward trend.

The graph for 2013 is tiny because of the buttdevastation brought upon by the Draven rally, which reduced average scores a lot. This is an interesting one because while the winner took 20%, most people below him took risks close to 10%. Apparently he was one of two people to pick Samus to go the finals. Most people had Mario, but Mario had that legendary defeat to Vivi in an early round. In the end, a single moment of inspiration brought him above the masses.

In 2015, most people who did well had below 15%. A lot of people went below 10% though, and the contest was not kind to those. The winner took 13% risks, so a healthy amount of risk taking paid off in the end.

2017 was our ultimate cookie contest, with the winner only taking a single risk (which was 3% of the total matches). Risks below 10% are the norm torwards the top, though, so this can be interpreted as an effect of the contest's uniqueness.

Finally, 2018 shows some clunky trends torwards the end, but generally, between 10-15% was the path to victory. The winner took 9%, though.

Conclusions:

  • Hypothesis 1 is right. If you want to win, it's probably a bad idea to differ from the masses by more than 20%, unless you really think you've figured something out the rest didn't.
  • The average risk of the guru winner is 14%, with most winners of 1v1 contests staying between 10 and 15%. Multi-ways actually seem to influence this ratio though, and it's probably better to take between 15% and 20% risks in them, although both in 2008 and 2009, the winner shot way above/below that.
  • Bracket size doesn't seem to matter much to risk taking, since both 2006 contests were small but had similar risk ratios to the rest. But contest type does. A more gimmicky contest, like Rivals or Villains, can give the win to more chalky brackets if there aren't many unpredictable matches. If such a contest is going on, watch where the pack is going and don't try to be a hero. This could happen again if we have a Best Console contest, for example.
  • It's still hard to know if the 10-15% range actually increases winning chance or if it's just the range most people are likely to be in anyway. Yet, there were a bunch of cases were the masses were either way above or way below that range, and the winner was still the guy who went closer to it (2008 is the notable exception). This gives some strength to the theory that you need a certain level of risks to avoid elimination, yet shouldn't go too much above 15% (or 20% for multi-ways). If you are lazy and don't want to think much, just take 14% risks. For this contest, this means 17-18 risks.
  • There is a lot of talk going about how this year's bracket seems chalky, yet we don't have much data so we could be wrong on a lot of things. The GotD1 results seem to match that sentiment. Expect to make a lot of mistakes, and yet, the 10%-15% range is likely to be the right one anyway. This means you should mostly follow the pack, but knowing which upsets to pick is critical. There will likely be some insane results no one will see coming. If you can see them, the contest could be yours.
  • One big limitation of this analysis is that it treats all matches as if they are equal, when truth is that later matches count more points (although the importance is not necessarily directly proportional to points, since you need to get certain early matches right to even have the right characters in the later ones). Ideally, I would also analyse how many points were risked by each guru, not just individual matches. But that would require a lot more data crunching since this info is not available in the guru site. And I wanted to finish this protect some time before bracket lockdown, so I had to go with what I had.
  • Please submit your bracket to the BOP so we can reliably calculate our risks, thanks. Link: https://gamefaqs.gamespot.com/boards/8-gamefaqs-contests/78450828


And this is it. I think I've filled my nerdiness quota for the whole trimester, at least. Let me know what you think of this analysis!
... Copied to Clipboard!
SuperNiceDog
03/08/20 3:55:33 PM
#3:


great analysis!

As one of the former Guru winners(Rivals 2011), I am happy to be a statistic.

---
Advokaiser
... Copied to Clipboard!
ZeldaTPLink
03/09/20 4:44:06 PM
#4:


Bump.

Also adding better versions of the graphs, which will go to the wiki later since I can't edit the first post. These ones actually show the guru winners, marked by diamonds in front of the curves.

This helps show how the winner eventually falls in the same rage (10-15% for 1v1 contests, 15-20% for multi-ways) even if the other contestants are all over the map.





... Copied to Clipboard!
GildedFool
03/09/20 5:23:09 PM
#5:


... Copied to Clipboard!
ZenOfThunder
03/09/20 6:25:50 PM
#6:


Whoa, super cool! We have the best community

---
(|| ' ' ||) drooling while eating
. /|_|\
... Copied to Clipboard!
#7
Post #7 was unavailable or deleted.
Whiskey_Nick
03/09/20 7:28:23 PM
#8:


Fantastic stuff here

---
I am Nick. Go Sens, Bills, Blue Jays! Advokaiser is a Guru
UotY 2015, You should listen to The Show w/ Ngamer and Yoblazer
... Copied to Clipboard!
SwiftyDC
03/09/20 7:46:12 PM
#9:


Tag.

---
dilateDChemist
... Copied to Clipboard!
Ngamer64
03/09/20 8:03:55 PM
#10:


Very cool stuff, glad I could help make this possible! I agree BTW, we've been lulled into chalk talk by SB's strong seeding when in reality we're going to be blindsided many times this bracket, just like we were 10 years ago.


---
Advo bested even The Show hosts in 2018, yowza!
board8.wikia.com | thengamer.com/xstats
... Copied to Clipboard!
ZenOfThunder
03/09/20 8:38:37 PM
#11:


this kicks ass

---
(|| ' ' ||) drooling while eating
. /|_|\
... Copied to Clipboard!
Leonhart4
03/09/20 8:54:35 PM
#12:


... Copied to Clipboard!
ZeldaTPLink
03/09/20 9:45:47 PM
#13:


Thanks guys! This was pretty fun.

That said, it's still statistics so maybe all of this is confirmation bias. But I feel like there is enough of a pattern here.

If anyone can explain to me what the hell happened in 2008 I'll be thankful.
... Copied to Clipboard!
squexa
03/09/20 10:01:59 PM
#14:


2008's wonky because of Snake.

Most people thought Link > Cloud > Snake but because of 4-way shenanigans, Snake beat Cloud twice and thus elevated a small group of people who made this correct call to the top. Some of them took a bunch of risks in the beginning that were possibly wrong but those early risks were worth much fewer points.

The thing to keep in mind is that not all risks are the same. Taking a round 1 upset is different than a semifinal/final upset.

---
congrats to BKSheikah the BYIG guru
... Copied to Clipboard!
shane15
03/11/20 6:41:32 AM
#15:


If someone gets a perfect D4 they are in with a shout.

---
... Copied to Clipboard!
junk_funk
03/12/20 5:21:15 PM
#16:


This is amazing. Thankyou.

---
Advokaiser came, saw, conquered.
... Copied to Clipboard!
chocoboslayer
03/13/20 8:38:15 PM
#17:


Yup, I'm here for this.

---
PotD's OFFICIAL King of Games
The Choco. The Cream of Guru Champ Advokaiser Fanboyism
... Copied to Clipboard!
theawesomestevr
03/13/20 9:31:55 PM
#18:


ZeldaTPLink posted...
In case you don't know, a "risk" is when you make a pick that goes against the board consensus. If most people take Link to beat Cloud, and you take Cloud, that's a risk.

So just to be clear, this was only an analysis of how many picks varied from the "cookie" and doesn't take into account the difference between taking an upset in the finals vs. only taking an upset in round 1?
... Copied to Clipboard!
ZeldaTPLink
03/13/20 9:47:01 PM
#19:


Exactly. See below.

ZeldaTPLink posted...
One big limitation of this analysis is that it treats all matches as if they are equal, when truth is that later matches count more points (although the importance is not necessarily directly proportional to points, since you need to get certain early matches right to even have the right characters in the later ones). Ideally, I would also analyse how many points were risked by each guru, not just individual matches. But that would require a lot more data crunching since this info is not available in the guru site. And I wanted to finish this protect some time before bracket lockdown, so I had to go with what I had.


... Copied to Clipboard!
GildedFool
03/15/20 4:28:24 AM
#21:


MetalmindStats posted...


separating out obvious matches where the guru winner took a risk just for the sake of it.
...don't do this.

---
... Copied to Clipboard!
MetalmindStats
03/15/20 4:35:11 AM
#22:


GildedFool posted...
...don't do this.
I'm guessing you think I classified certain matches where the winner was different than B8 expected as chalk matches, or something along those lines? (Hint: I did no such thing.)

If not, please clarify your comment.

---
"I believe in a universe that doesn't care, and people who do."
You won the CBX Guru Contest, Advokaiser! Bully for you!
... Copied to Clipboard!
GildedFool
03/15/20 5:58:28 AM
#23:


Do not shape, alter or edit data in a way that you think makes it better and then perform any sort of analysis on that data.

---
... Copied to Clipboard!
MetalmindStats
03/15/20 6:23:11 AM
#24:


I concur with everyone else: this is a great analysis!

I previously made a post of my own about guru winners predictions relative to chalk, hence the posts above, but I decided to delete that post.

---
"I believe in a universe that doesn't care, and people who do."
You won the CBX Guru Contest, Advokaiser! Bully for you!
... Copied to Clipboard!
NewerShadow
03/16/20 12:29:43 AM
#25:


ZeldaTPLink posted...
Exactly. See below.

Since I have the data handy (and need to brush up on my codebase for the new contest), here's a set for the last contest:

List of risks (differences from the cookie): https://raw.githubusercontent.com/andrew-howes/Guru2k18/master/cookieOUT.txt
Just a list of User/difference scores: https://raw.githubusercontent.com/andrew-howes/Guru2k18/master/cookieOUT2.txt
... Copied to Clipboard!
ZeldaTPLink
03/16/20 6:38:16 AM
#26:


NewerShadow posted...
Since I have the data handy (and need to brush up on my codebase for the new contest), here's a set for the last contest:

List of risks (differences from the cookie): https://raw.githubusercontent.com/andrew-howes/Guru2k18/master/cookieOUT.txt
Just a list of User/difference scores: https://raw.githubusercontent.com/andrew-howes/Guru2k18/master/cookieOUT2.txt

Cool.

Yeah I can make a point-related version of this but it's not coming out before bracket lockdown, I'm afraid.
... Copied to Clipboard!
hylianknight3
03/21/20 5:39:06 AM
#27:


Very cool and excellent analysis!

---
Kampfmaschine Advokaiser punched my lights out in the 2018 Guru.
My bracket was soft... just like my heart!
... Copied to Clipboard!
Glenn_and_Toad
03/23/20 2:39:31 PM
#28:


Just want to say I love this topic.

---
SC2K13 Bracket: Kefka hate hate hate hate hates my bracket!
Oracle: Team Frog and Magus
... Copied to Clipboard!
ZeldaTPLink
03/24/20 10:35:17 AM
#29:


Thanks for the appreciation, guys.

I need to remember to archive this soon.
... Copied to Clipboard!
Advokaiser
03/27/20 2:55:01 PM
#30:


Oh hey, I'm in a graph!

---
It's an honor being part of Board 8's community.
... Copied to Clipboard!
pjbasis
03/27/20 4:06:54 PM
#31:


... Copied to Clipboard!
TsunamiXXVIII
03/27/20 7:58:06 PM
#32:


squexa posted...
2008's wonky because of Snake.

Most people thought Link > Cloud > Snake but because of 4-way shenanigans, Snake beat Cloud twice and thus elevated a small group of people who made this correct call to the top. Some of them took a bunch of risks in the beginning that were possibly wrong but those early risks were worth much fewer points.

The thing to keep in mind is that not all risks are the same. Taking a round 1 upset is different than a semifinal/final upset.

It wasn't just Snake. Vincent had knocked Crono out of the contest in 2007, becoming the first character to ever beat a Noble Niner in a Contest Battle (L-Block did finish second to Snake in their R3 meeting and the Vincent > Crono was earlier in R4 than L-Block's matches), so he was favored to do so again in 2008, and furthermore the cookie then had Vincent taking advantage of Link's dominating SFF to advance past Mario and Samus in R5--which is exactly what Crono did.

And those are just the ones that the Guru winner got right. The Cookie also expected Kirby to advance in second place in Round 2, behind Master Chief, before going out in Round 3 behind Master Chief/Dante. Instead, it was Master Chief who finish second in R2 before going out in R3 behind Dante, while Kirby added an upset of Sonic in R4 for good measure. This was one that the winner got wrong, though it does help to explain why the runner-up had so many risks as well; he got it, well, a lot closer to right. Besides, absolutely no one correctly called Kirby getting past R4 so correctly calling him finishing in first place the previous two rounds was the best anyone did. Said runner-up also correctly called that the fourth character in the Samus/Vincent/Crono match would not be Ryu, but Pikachu.

And that doesn't even account for Weighted Companion Cube reaching Round 4, which only one Guru called correctly but they were a little too reliant on the jokes and had L-Block returning to the final. Or both Square characters advancing out of the Squall/Yoshi/Sora/Fox match in R2, giving Mewtwo a free pass to R4 in the wake of Cloud's SFF hammer (cookie expectation was that Yoshi would beat out Sora for second place behind Squall, then ride Cloud's SFF to get past Squall in R3.)

The winner might have been Link as usual, but yeah, 2008 had a lot of unexpected results along the way.

---
Also known as Cyberchao X.
... Copied to Clipboard!
Axem_Turtle
03/28/20 1:21:29 PM
#33:


You touched on it, but for what it's worth, GotD, I won based mainly on 2 late game upsets that almost noone else had, Majora's Mask and Brawl over Melee. A good chunk of the board had GSC beating Majora, so I was able to get huge points from that and the following rounds. I would have won the whole contest if Majora lost to FFX
... Copied to Clipboard!
Axem_Turtle
03/28/20 1:24:00 PM
#34:


Could you "scale" the risk % by normalizing against the number of points available for each match? That would give a higher weighting to the bigger risks
... Copied to Clipboard!
ZeldaTPLink
03/28/20 1:37:41 PM
#35:


Axem_Turtle posted...
Could you "scale" the risk % by normalizing against the number of points available for each match? That would give a higher weighting to the bigger risks

I can but... that's one hell of a data crunching to do. Probably not going to do it soon, since this project was supposed to be finished before this contest's bracket lockdown.
... Copied to Clipboard!
ZeldaTPLink
03/28/20 1:41:48 PM
#36:


Basically I need to get the raw picks file from NG for every contest, run them through a program that extracts the match results, calculate the cookies (I've done that for Series 2006, but it was manual, I need to figure out a way to automate it for the rest), then calculate the risks. And only then I can make the graphs.
... Copied to Clipboard!
ZeldaTPLink
04/03/20 11:09:13 AM
#37:


... Copied to Clipboard!
ZeldaTPLink
04/03/20 11:12:53 AM
#38:


Or I thought I had, but the images glitched. Let me fix that;
... Copied to Clipboard!
ZeldaTPLink
04/03/20 11:23:44 AM
#39:


Ok this doesn't look as smooth but I at least managed to leave the images there as thumbnails.
... Copied to Clipboard!
HaRRicH
04/07/20 11:29:55 PM
#40:


Just now saw this -- great analysis! This is a good reason to love these contests.

---
... Copied to Clipboard!
Topic List
Page List: 1