An apology to those of you on the "less dorky" side of my blog readership, because today is going to be a bit dorky and math-y. Well, OK, a lot dorky.
Anyway, this is something that's been on my mind a lot because I play a good bit of this online card game called Hearthstone, made by a big gaming company called Blizzard. Any time you build an online game that people can play against each other, you end up running up against one of these game design problems that's particularly thorny: how to match people up to play against each other. Of course, you can always play against your friends, or people you know personally, but generally, at some point you expect the designers of the game to hand you someone fun to play against that you don't know. A general design principle of fun is to make something challenging, but not too challenging. And that applies here: it's generally fun to play against people who are about as good as you are; sometimes slightly worse, sometimes slightly better. The question is how to make that happen, and in an automated way.
A lot of smart people have thought about this problem, and since long before computer games. One of those was a man named Elo, who was a top chess player. He designed a system which you can read about here: https://en.wikipedia.org/wiki/Elo_rating_system. The essence of his system is that every player is assigned a number. Everyone starts at 1000, and you go up and down based on how you do in matches against other players. If you beat someone, you get points, and if you lose, you lose points. How many points you gain or lose is based on how much higher or lower they are than you. So if you are at 1000 and you beat someone who is at 1500, you get a lot of points, maybe 15 or 20 or even more. If, on the other hand, you beat someone who is 900, you may only get a half a point - or no points. There are lots of variations on Elo's scheme, and some of them are in active use in online gaming. They work reasonably well in some ways, and not as well in others. And Blizzard decided that wasn't good enough.
A quick note about Blizzard: they are an incredibly successful gaming company. Along with maybe Valve and Nintendo, they are household names among gamers. In particular, Blizzard is known for building online multiplayer games. That's their thing. Another thing about Blizzard: they consider themselves design geniuses (and with good reason). And their main design goal is, and always will be, fun. They want everyone playing their games having fun, all the time. And one problem with Elo is: it isn't really very fun. It's very math-y. It's kind of hard to explain. When you win or lose it can be really hard to tell what just happened. And it's kind of relentless; depending on the implementation, you find that you're always playing someone slightly better than you. This is great if you're a dedicated chess champion who wants to hone their craft; it's less good if you're on your lunch break and just wanted to have a quick fun game.
So, in particular, for Hearthstone, Blizzard implemented a sort of hybrid system. For really really good players - what they refer to as Legend - they use a system that's close to Elo. But for everyone else, they use a much simpler system. Basically, when you lose, you lose a point, and when you win, you get a point. Then, you are generally matched against players that have about as many points as you do, plus or minus a couple. That's it. Simple, right? And pretty effective, at least on paper. After all, people who are better win more, and get more points, and so they play against harder players. And everyone understands what's going on.
Except that it has a few flaws. And those flaws aren't really obvious until you've been playing for a while. For myself, I noticed that sometimes, I seemed to go on these long winning or losing streaks. Even though I always had about 20 points or so, sometimes the players I played against just seemed, well, better than other times. At first I thought I was being paranoid, but finally I figured it out: I was playing at different times of the day.
One of the things about an Elo system which has always been a theoretical problem is the idea of disjoint sets. That is, sets of players who never play against each other. Imagine, for example, that a high school chess club sets up an Elo system and starts to play against each other. Their Elo scores will rise and fall as they play, and they may begin to accurately reflect their status inside that club, but they will never be meaningful when compared against scores of anyone else. In the real world of chess, that's a serious problem; serious enough that different systems were designed to combat it, such as only getting points when you play against known chess masters. But in the world of digital combat, such a scenario seems really unlikely. Why would any such set of players evolve? The world is so big, and so heterogenous, that eventually the graph of players that play each other will join up. And, indeed, if Hearthstone used a true Elo system, that would happen. But it doesn't; it uses this modified system. And in this modified system, there is no way to significantly change a person's rank based on one (or a small number) of games. That means that if I only rarely play anyone outside my own small set, that the effect can't be enough to "join up" the graph. But - again - why would anyone consistently play other players that only come from a certain set? Well, one way that could happen is geographically; if I'm paired with players based on geography - and I usually am, for technical reasons such as network speed - then my score is really only safe to compare inside that geographic zone. Historically, that hasn't been a huge deal because, unless you're a racist, there's no reason to think that players in North America, or California, are better than players in Ireland.
But time of day is different. It appears - and this makes sense - that the demographic makeup of players at 10 in the morning PST is meaningfully different than the players at 5 pm. I'm not sure exactly why this is, although I could speculate. But there is a sizable and noticeable jump in player quality - for the same rank - at about 5 pm PST. My guess is that this represents about 8 pm EST, which is when the best players, who are often adults, get home from work and log on to play. They play for 3 or four hours, then they go to bed. So if you play any time outside that window, you never play against them.
I have no idea if this is true. It's merely subjective evidence. But I can demonstrate, mathematically, that this is at least a theoretical problem with the system. And I can say that subjectively, this theoretical problem appears to be a real issue.
How to fix this? Well, they could use a true Elo system, of course. Failing that, they could use Elo as a backup. Meaning, when they go to pair two players of rank 20, they could also use their (hidden) true Elo scores to make the best pairings. This would require a large pool of players, but Hearthstone has a large pool of players. Perhaps they already do this, although if they did, I suspect I wouldn't be noticing the issue.
