On Variance

Today's Soundtrack: https://open.spotify.com/track/6Ez7j7MPocvayb7J7WlaR7?si=7c2e6478e38c4b86

Magic twitter was on fire last week with discussions about whether there is increased variance in limited and whether this means that skill matters less these days. My position, which I've of course tweeted about, is roughly yes to the first, no to the second. I thought I'd write today about this classic Magic topic, and how I've learned to approach it over the years.

Defining Variance

Variance is the perfect topic to get people riled up because it's pesudo-logical, emotional, and ill-defined. Let me explain what I mean by each of those terms.

Pseudo-logical

Variance is, at least theoretically, a mathematical term. There's an equation for it!

Of course, this equation doesn't really give us an intuition for what it's trying to calculate.

Variance is the opposite of consistency in a range of outcomes or values. High variance means that the universe of possibilities is wide and highly unstable; low variance means it is narrow and relatively stable.

But to talk about variance we need to define what our outcomes or values actually are. And this is where the pseudo-logic comes in when we turn to discussions about variance in Magic. I'm not using pseudo-logic as a pejorative term; it's necessary to use intuition rather than hard calculation to discuss these things because Magic is such a complex game. If we had a formula to calculate variance in Magic, we'd very likely be able to design a computer program to optimize for it. Nobody wants that.

Ill-defined

When people talk about 'variance' in Magic, they often end up conflating different outcomes and values. Here are some of the things people are actually talking about when they talk about 'variance':

• Opening hands/mulligans

• Lucky or unlucky draws that decide the game in a visible way

• Random outcomes determined by cards (think flipping coins, partial deck searching, etc)

• Hidden information and opponent behavior

• Rare quality in limited

• Synergy availability in limited

• Pool strength in limited

• Deck strength in limited

• Pairings/matchups in constructed

• Tiebreakers in tournaments

• Match winrate

• Tournament winrate

• Travel interruptions, health, and other meta-tournament factors

• Etc, etc, etc, etc....

There's a lot here. What should be the ultimate 'outcome' we care about? This is, obviously, a contextual and subjective choice, but for today I want to focus on a combination of "Match winrate" and "Tournament winrate" I'll call "Tournament success rate", roughly, the rate at which a given player will achieve a high-level finish at a tournament that satisfies their goals for that tournament.

Emotional

I'm choosing Tournament Success Rate (TSR) as the 'outcome' we care about measuring variance over because I believe it is the outcome that matters most personally to every Magic player. We care about our successes at tournaments, a LOT. I'd argue it's what most competitive players play the game for. The rush one gets from completing a deep run at a tournament, proving themselves and feeling like they conquered a field of other competitors with superior skill is an incredible high.

It's also a dangerous one to chase, from an emotional perspective, because pretty much everyone will admit that the expected outcome for nearly every Magic player at nearly every tournament (Simon, Jason, Javier, I'm not talking about you) is failure to reach your goal.

This is okay!!! Most things worth doing are worth doing because they are hard. Succeeding at Magic tournaments is extremely hard, so it makes sense that the feeling you get when you do that should be extremely worth it.

Here's the issue, though. Success at Magic tournaments is not completely about merit. It is, always, extremely possible for the best player in the room to lose every match, and possible for the worst player in the room to go undefeated.

This is both a blessing and a curse. Most people tend to focus on the 'curse' part, because it means that no amount of edge, no amount of preparation, no amount of sheer willpower and brainpower can overcome the possibility of failure. It is simply impossible to will your way to winning a tournament, despite the emotional lie that many content creators over the years have suggested. I am personally willing to believe that in 99% of reasonably sized Magic tournaments ever played, the winner of that tournament got luckier, overall, than they would have expected to be on that day. It is almost impossible to succeed a tournament without being at least a little bit lucky, which means that it is always possible you will not win the tournament because you didn't get lucky enough.

This would be an emotionally difficult enough fact to deal with, but Magic takes it a step further. Did you know it's nearly impossible to quantify how lucky or unlucky someone got (in terms of TSR) in a game or tournament?

I mean, of course you know this intellectually, but that doesn't stop all of us from trying to answer that question all the time, constantly. You try to answer this question about every tournament you play in without even thinking about it.

How often is your answer that you got lucky? How often is your answer that you got unlucky?

I have been playing Magic for 15 years. I can tell you that the ratio of my explanations is still something like 20/80, even though I know, logically, it must be extremely close to 50/50 for my entire Magic history. I think most people are probably at something like 5/95.

The logic here is extremely easy, but the emotions are extremely powerful - and our brains, yes, even nerdy, "logical" competitive Magic players' brains who are trained and incentivized to think rationally about variance, are ultimately slaves to our emotional needs. We need to be able to take credit for our wins and explain our losses with things other than "I could have done better, and I didn't".

The Things I Don't Want to Say

So we know that variance is pseudo-logical and emotional. We've tried to address the ill-defined part, but we're still left with a phenomenon that deeply affects an outcome we care strongly about (TSR), and is extremely difficult to understand.

Let me first try and get out of the way the answers that I think are unhelpful. I don't want this piece to be just another one of those things where a pro lectures at you that variance is a natural part of the game, and if you were really smart, you wouldn't care about it at all.

It's easy to say we shouldn't care about variance, but the fact is that Magic is interesting because of its built in variance. Accurately understanding how variance works is crucial to actually succeeding at the game. In Magic, you constantly need to make decisions based on a fuzzy notion of how likely certain ranges of outcomes are. Even something as seemingly "emotionally based" as worrying about how likely you are to win the game you're in can matter, because it can be important to play more aggressively when you're behind and more conservatively when you're ahead.

Simultaneously, I also don't want this piece to be a defense of leaning into your emotions around variance. Magic is a game about focusing on what you can control and focusing on what matters. Variance matters, and "controlling it" to the greatest extent possible in the long run is pretty much synonymous with becoming an elite player, the paradox is that there is a certain point after which variance falls firmly and irrevocably into the category of things we cannot control.

Unfortunately, so many of the things we naturally want to invest our emotions in around Magic are on the "uncontrollable" side of that devilish coin. Once it's out of your hands, it's out of your hands, and if it's out of your hands, we'd better work on getting it out of your head as well.

Zooming out

I'm coming back to write the second part of this after finishing a sealed tournament on MTGO, one where the way my draws went in the final game had me begging Mora to listen to my bad beat story and watch the game replay as soon as they could.

I say this just to show that I am far, very far, from immune to letting swings of what feels like bad luck mess with my emotions in Magic. But over the years I have developed tools to re-center on the controllables. The biggest one is zooming out.

Remember that huge list of possible definitions of 'outcomes' there can be variance in in Magic? All of these outcomes can, and do, end up influencing TSR. You could even say that TSR is an aggregation of these smaller outcomes. My success in a sealed tournament is a combination of my pool's rare quality, its synergy availability, its mana flexibility, my opening hands, my lategame draws, cards that produce random outcomes in the game, the way that hidden information gets semi-randomly fed to me throughout the game, and of course, all the same factors for my opponents.

If you're looking for reasons you lose in Magic because of bad luck, you'll find them. It's really not that hard. There are simply so many outcomes that collectively make up tournament success, that you are bound to get unlucky in certain places. If you lose, it is almost guaranteed that you could have won if a particular very unlikely thing didn't happen.

Let me say that again, because while it can sound trivial, it's something I think not many people actually grasp on an intuitive level.

If you lose a game or have a bad result in a tournament, it is almost always true that you could have won, but some very unlikely thing happened that made you lose.

More bad news: This only gets more true as you get better at the game! The best players in the world almost always lose games where they got quite unlucky in some way or another, because they win all the games where that doesn't happen to them (and still win some where it does).

It might seem, then, like Magic is just a minefield of swingy outcomes, where variance determines everything. After all, if the ultimate result we care about is made up of a bunch of highly swingy factors, it intuitively feels like something like TSR must be incredibly swingy.

... and it is!

... but not exactly.

This is the paradox of variance. It turns out that when you combine many high-variance outcomes into a single value, that value is actually significantly less unpredictable. This is obvious in a way - It is a simple mathematical truth that the result of 100 coin flips is a lot less "random" than that a single coin flip. The counterweight, of course, to the near-certainty that you will get unlucky in certain ways is that you will also get lucky in many. More chances for both things to happen leads to a higher likelihood that things will even out in aggregate, while being swingy at small scales.

There will always be negatives to focus on, if that's what you choose to do. They won't be fake, they will be genuine. You will get incredibly unlucky over, and over, and over again, while you are, in the lower-emotional-payoff background, also getting lucky.

I really want people to sit with this.

Magic tournament success is determined by a long list of high-variance factors. This means it is always possible to have extreme swings of "luck", but it also means that over the long term, tournament results will be more predictable and "stable" than the factors that determine them.

The data we have shows this. We know that certain players have consistently high (but not 100%) winrates over long sample sizes. I've talked about Eken before, but he is really the archetypal example of how consistently it is possible to win in Magic.

Past performance in Magic simply is predictive of future performance if the sample sizes are sufficiently high, and we do have large sample sizes in some contexts. Those sample sizes tell us that some players are vastly more successful over long periods than others - and, of course, that those players still do lose somewhat often.

It's easy to feel the intensity of variance in the small details, so zooming out is really important to maintain perspective. I want everyone reading this to try on this statement and see how it feels:

My historical tournament performance in Magic has been very, very close to a "fair" outcome for my skill at the game.

This statement is true for almost everyone. Believing it is true for you is the most rational and accurate belief you can have.

Can you accept it emotionally? How does it feel? Is it difficult?

Now how does it feel that you mulliganed to 5 in your last game, kept a 2-lander, and didn't see another land until turn 5?

At least for me, zooming out does help me grapple with the moments where I feel like I've gotten incredibly unlucky. It reminds me that I've had lots of successes and lots of failures, but at the end of the day, the aggregate is fair. That hurts a little bit, but accepting it allows me to refocus on what matters - my own process.

Your Bad Draws Didn't Fall out of a Coconut Tree 🥥

Admonishing people not to focus on the times they get unlucky is, in my opinion, an ineffective strategy for getting better at Magic. When our brains are trying to quantify exactly how unlucky we just got, they are actually trying to get better at Magic. Developing a strong probabilistic intuition is a critical skill. It actually really does matter how unlucky you just got! Understanding this accurately can make a difference in the lessons you take from certain games.

The thing we want to do is to balance this analysis with a more holistic one that takes into account as many factors as possible.

The sealed game I just lost had a moment where if you paused the game, it was nearly inconceivable that I would lose. The sequence of draws I and my opponent would have to have for them to come out on top would last across 10 more turns, and at every stage they needed to both draw the right answers for my threats, and have me draw almost nothing while that was happening. It was a slow, slow, slow, SLOW march from 95% chance at victory to 0%, and the entire time I had almost no decisions to make. It was quite frustrating! After reviewing the game with Mora, they evaluated that I had made zero mistakes - I had just gotten extremely unlucky.

Okay, so we've established that I got unlucky in that game. Or more precisely, between turns ~6-16. But let's zoom out. To get into that position in the game, I had to have a pretty strong draw for my deck, and my opponent had to have a relatively clunky draw. We're already somewhere in the luckiest 25% of outcomes for how a game could start for me. So even when you see me as unlucky, all of that bad luck came in a game that looked the way it did because of my luck.

Still, my deck was probably better than my opponents', so I had earned that advantage to some extent. Except, I do think my pool was significantly better than average... so that's another point of luck.

I also had shifted my deck to be a more consistent, but less fundamentally powerful build for games 2&3 (I cut a splashed bomb and added in some medium creatures). This choice will tend to lead to more consistent starts, but slightly less power as the game goes long. I knew my opponent had a sweeper in their deck, and I was mostly making this swap because it had felt like the stronger overall deck, and I had had some consistency issues in previous rounds. Looking back, the slower, more grindy deck that was admittedly less consistent may have been better positioned against the deck with a wrath. I may have taken the wrong lesson from the previous rounds, which were, after all, a pretty small sample size. It might have been better to conclude that my initial build was fundamentally sound, and I had indeed just gotten somewhat unluckier than I should expect before.

This is how thinking about variance in Magic works when you are doing it effectively. The more factors you consider, the more opportunities there are to find mistakes, and the more difficult it is to really say whether you overall got lucky or unlucky -- because, in the end, it's usually a mix of both that is closer to fair than it seems.

I went 2-2 in this prelim, and generally have been going about .500 in them recently. Is that fair? Probably. I'm an extremely good limited player, but these prelims are generally just 12-20 of the best MTGO limited grinders, so it's not like I should be expecting to 4-0 every time. I'm hoping that playing in more of these will challenge me enough to grow my game, and eventually maybe I'll be able to win at something like a 60-65% clip like I do on Arena. But for now, the results are fair. I don't think I'm getting very unlucky.

Seeing the big-picture of variance accurately allows us to feel our frustrations with the small-scale swings, but still spend most of our mental energy thinking about our controllable decisions, and finding places where we're possibly still making mistakes in predicting outcomes. The solution to our emotional, pattern-matching brain is not to disregard or numb ourselves to frustrating outcomes, but instead to place them in the context of all the other outcomes in which they exist and that which came before.

I'll leave everyone with this image, which in my opinion accurately captures what it feels like to battle in the sealed trenches: