Games aren’t much fun if you can’t play them, so naturally struggling to figure out the rules can be one of the most frustrating parts of boardgaming.  Sometimes the error is on the part of us players, but in my experience, designers and publishers deserve the lion’s share of the blame. Worse, as more and more publishers “shotgun” publish games, pushing them quickly out the door to meet deadlines, the problem is getting more noticeable.  The Spiel des Jahres jury observed that they had to eliminate a record number of games from award contention in 2018 because of poor rulebooks.  

I was recently chatting with some fellow game reviewers about Charterstone, a game I gave a negative review after struggling to figure out how to even play parts of it.  They seemed skeptical of my criticism, so I pointed out that, despite it having only about 5,600 ratings on BGG, it already had more than 740 rules threads.  That’s shockingly bad: there’s a rules thread for about every 7.5 ratings.

I’ve long loved to think about data in board games, and I’ve gotten better at pulling BGG data, so I started gathering as much info as I could about rule quality.  I call the metric I adopted the “Rule Quality Index” (RQI), mostly because I needed a shorthand to refer to the calculation in this article.  RQI is simply the number of ratings a board game has divided by the number of rules threads a game has inspired. It’s a crude way to evaluate the problem, but it’s the best method I could think of.  

What follows is my attempt at empirically evaluating rule quality among the BGG top 100, plus some other categories of games.  This is my effort at shedding some light on what I see as a growing problem.

THE METHODOLOGY, AND SOME CAVEATS

To calculate the RQI of a game, you just need two numbers.  First, from the stats tab for a BGG game, pull the total number of ratings.  Second, from the forums tab, pull the total number of rules threads. Just divide the first number by the second to get the RQI.  A lower score is bad; a higher score is better.

I admit that it is a crude measure of rule quality, but it does allow you to compare, across games, how many players felt bothered enough to create a rule thread in relation to the number of BGG users that actually rated the game.  That’s the single best metric I can think of using publicly-available data.

Using the number of ratings isn’t perfect.  Not everybody rates games, plus there are countless other reasons not worth exploring here.  But as a preliminary matter, the only other metric I considered worthwhile was the number of owners of a game, which is also available on the BGG stats tab.  For the first several games, I looked at my calculation both ways, and though they yield a different result in an absolute sense (meaning the RQI is a different number based on whether you use owners or ratings), relative to other games, it didn’t make much difference.  Simply put, it doesn’t matter because the number of ratings of a game tends to be correlated with the number of owners. Given that there wasn’t a relative difference, I went with ratings, as it seems to me people would be more likely to ask a rules question if they’ve played a game than if they’ve merely bought it.  

Using the number of rules threads also isn’t perfect; it’s simply the best stand in I could think of.  If you spend a few minutes glancing through the rules threads, many of them are duplicate, and a few of the questions are actually strategy questions with little relation to the rules.  And, if we’re being really blunt, some of the rule questions are downright unnecessary. There are 114 rule threads for Ticket to Ride, but if you’ve ever played Ticket to Ride, you know that there simply shouldn’t be 114 different rules threads, as it is a decently intuitive game.  (Nonetheless, Ticket to Ride has a very high RQI because of the sheer number of ratings.)

The final large caveat is that, while in general rulebooks are leading to issues with rules quality, and while I usually refer to rulebooks in this article, they aren’t the only components that contribute to rules questions.  In some games, the rules are on the non-rulebook components, especially cards, and that can also lead to problems. In fact, more often that precise issue seems to be exactly what led to problems: games with heavy text on cards appeared, at first glance, to be a repeat offender on the lists below.  

Here are some other rapid fire issues I admit to having with my own analysis:

• Rules issues are decently correlated with game weight.  But none of us really know what game weight means, since it means different things to different people, so I don’t entirely know what to make of that correlation.  I can tell you that there are a number of games with a high (i.e. good) RQI that are decently heavy according to their BGG weight, a fact which surprised me.
• I made no attempt to deal with the fact that games get new editions with updated rules (reducing the ambiguity/vagueness problem), nor the fact that rulesets might be better drafted in different languages.  Frankly, I think it is impossible to deal with either issue.
• Some people ask questions in the base game threads even if the questions pertains to an expansion, so games with expansions might be at a slight disadvantage.  
• Games with very few ratings tend to have a bad RQI out of the gate, since people ask one or two rules questions before there are many ratings.  Often BGG users ask rules questions before the rulebook is even published. To avoid this, I basically didn’t look at games with a low number of ratings, so almost everything in my analysis has at least 1,000 ratings.  
• Some ratings may be artificially high when spread across multiple sites.  A friend pointed out that Magic: The Gathering sits at 57.5, which is above median, but also observed that most Magic questions would go to specific forums and not BGG.

There are other issues I’ll address below with specific games, or specific types of games.  

And I’m sure there are some issues I didn’t even bring up.  But remember, this is a light-hearted, non-scientific analysis, and if you find yourself writing a snarky comment about how I didn’t consider this factor or that factor, or worse, if you start talking about peer review, I can already tell you my response: go write your own analysis, and I’ll happily read it.  

BRING ON THE DATA…

I ran the numbers for every single BGG top 100 game.  The numbers were run on August 29, 2018. To provide a frame of reference for what follows, the median RQI in the Top 100 is a 45.6.  Keep that in mind when comparing games.

Here are the ten highest (i.e. best) RQIs, from highest to lowest.  These are the games that should be rewarded.

BGG Top 100 – 10 BEST RQIs	Threads	Ratings	RQI	Weight
Codenames	70	44,130	630.4	1.3
Ticket to Ride: Europe	83	43,221	520.7	2.0
Patchwork	73	28,757	393.9	1.7
The Resistance: Avalon	59	20,358	345.1	1.8
Sherlock Holmes Consulting Detective	37	11,958	323.2	2.7
Azul	59	15,305	259.4	1.8
Dominion: Intrigue	112	25,825	230.6	2.4
Concordia	76	16,292	214.4	3.1
7 Wonders Duel	164	34,043	207.6	2.2
7 Wonders	338	63,089	186.7	2.3

As you can see looking at the BGG weight, lighter games do pretty well here.  But not all of the games are on the lighter side: Concordia in particular seems to stand out as a game with a high RQI but also an above-medium weight.  

It is notable how many of these games have won major game awards: there are 3 Spiel des Jahres winners in here (Azul, Codenames, Sherlock), 2 offshoots of SdJ winners (Dominion: Intrigue, Ticket to Ride: Europe), and 3 IGA winners (Ticket to Ride: Europe, 7 Wonders, 7 Wonders Duel).  Game award juries (rightfully) look at rule quality, and this shows that.

Now let’s look at the 10 worst offenders, which are arranged from lowest to highest.  These are the games that lead to a significant number of rule questions.

BGG Top 100 – 10 WORST OFFENDERS	Threads	Ratings	RQI	Weight
Kingdom Death: Monster	1,406	5,020	3.6	4.2
Gloomhaven	3,700	17,854	4.8	3.8
Twilight Imperium: Fourth Edition	553	3,777	6.8	4.1
War of the Ring (First Edition)	1,276	9,308	7.3	3.8
Descent: Journeys in the Dark (Second Edition)	2,410	17,878	7.4	3.2
Star Wars: Imperial Assault	1,873	15,057	8.0	3.3
Android: Netrunner	2,797	23,194	8.3	3.4
Arkham Horror: The Card Game	1,566	13,286	8.5	3.3
War of the Ring (Second Edition)	847	8,571	10.1	4.1
The 7th Continent	641	7,057	11.0	2.9

Apparently having lots of rules questions doesn’t stop a game from grabbing BGG’s top spot, as Gloomhaven made this list of shame.  (As I discuss below, legacy- or campaign-style games often have more rules threads than average.)

But the really noticeable part of this list is the Fantasy Flight problem: more than half the games on this list have a Fantasy Flight affiliation.  That company is known for their terrible rulebooks, and I suppose this is decent evidence of that. And that is especially true in light of the fact that some of these games are on later editions and they still have rulebook issues.  

As I alluded to before, games with a really high RQI tend to be lower-weight games, and games with a poor RQI tend to be heavy. But there are games that buck the trend both ways.  For example, here are the 10 best RQIs in the BGG 100 with a weight greater than 3.0.

BGG Top 100 – BEST RQIs WITH WEIGHT >3.0	Threads	Ratings	RQI	Weight
Concordia	76	16,292	214.4	3.1
The Castles of Burgundy	183	32,765	179.0	3.0
Puerto Rico	378	53,663	142.0	3.3
El Grande	159	20,296	127.6	3.1
Tzolk’in: The Mayan Calendar	181	22,044	121.8	3.6
Power Grid	469	47,418	101.1	3.3
Mombasa	74	7,432	100.4	3.9
Tigris & Euphrates	229	22,190	96.9	3.5
Alchemists	145	13,138	90.6	3.8
Caverna: The Cave Farmers	253	21,720	85.8	3.8

Looking at this list, massive props go to Czech Games Edition (CGE).  Not only did Codenames make the good list above, but Tzolk’in and Alchemists both made this list too.  CGE puts a lot of emphasis and thought into their rulebooks, and it shows here.

The above chart demonstrates also one of the most interesting finds I encountered.  While game weight was always going to be correlated with RQI, I was surprised by how weak the correlation was, and I think that is because rulebook quality is driving the questions, not necessarily the complexity or thinkiness of the game itself.  

This is demonstrated in this chart (with this link going to an interactive version) developed by Opinionated Gamer Luke Hedgren.  

IF YOU WANT YOUR GAME TO BE WILDLY SUCCESSFUL…

As I said earlier, the Spiel des Jahres jury noted this year that they had to eliminate a record number of games from award contention because of poor rulebooks.  Even among their heavier award category (the Kennerspiel des Jahres) all of the winners have very few rules threads:

KENNERSPIEL DES JAHRES WINNERS	Threads	Ratings	RQI	Weight
7 Wonders	338	63,089	186.7	2.3
Village	142	16,171	113.9	3.1
Legends of Andor	396	10,311	26.0	2.8
Istanbul	161	17,397	108.1	2.6
Broom Service	43	5,333	124.0	2.4
Isle of Skye	73	13,388	183.4	2.2
Exit: The Game – The Secret Lab	1	2,939	2939.0	2.5
Exit: The Game – The Abandoned Cabin	2	4,351	2175.5	2.6
Exit: The Game – The Pharaoh’s Tomb	1	2,772	2772.0	2.9

The median weight among the BGG top 100 may be a 45.6, but even among KsdJ winners, it is a 124 (counting all of the Exit Games as one).  Andor was reportedly better in its German version (the Fantasy Flight version has significant errata), but odds are Andor was always going to be lower because of the sheer amount of text.

And, as expected, the RQI is even higher with the lighter Spiel des Jahres.  Here are all of the Spiel des Jahres winners from Catan in 1995 to Azul in 2018.  The median here is a 184.8. (An enormous “thank you” to my friend Brandon Kempf for pulling this data.)

SPIEL DES JAHRES WINNER	Threads	Ratings	RQI	Weight
Azul	59	15,305	259.4	1.9
Kingdomino	39	14,750	378.2	1.2
Codenames	70	44,130	630.4	1.3
Colt Express	101	16,640	164.8	1.8
Camel Up	22	16,632	756.0	1.5
Hanabi	52	29,332	564.1	1.7
Kingdom Builder	129	15,679	121.5	2.1
Qwirkle	47	12,426	264.4	1.7
Dixit	19	37,927	1996.2	1.3
Dominion	591	64,203	108.6	2.4
Keltis	20	4,820	241.0	1.7
Zooloretto	73	12,571	172.2	1.9
Thurn and Taxis	102	16,188	158.7	2.3
Niagara	41	7,087	172.9	1.8
Ticket to Ride	114	56,303	493.9	1.9
Alhambra	142	23,949	168.7	2.1
Villa Paletti	23	3,674	159.7	1.2
Carcassonne	370	78,729	212.8	1.9
Torres	78	7,654	98.1	2.9
Tikal	103	15,370	149.2	2.8
Elfenland	17	7,171	421.8	2.2
Mississippi Queen	23	3,479	151.3	1.7
El Grande	159	20,229	127.2	3.1
Catan	402	79,127	196.8	2.4

A few months ago, we at The Opinionated Gamers published our list of 50 Modern Classics.  Those games almost all had high RQIs. The exception is Agricola, which likely had a lot of questions from specific cards.

#10 to #1 on Our 50 Modern Classics List	Threads	Ratings	RQI	Weight
#10 Agricola	1,300	54,295	41.8	3.6
#9 Power Grid	469	54,740	116.7	3.3
#8 Catan	402	79,127	196.8	2.4
#7 Carcassonne	370	78,705	212.7	1.9
#6 Ra	74	16,768	226.6	2.4
#5 Puerto Rico	378	53,663	142.0	3.3
#4 Princes of Florence	81	13,792	170.3	3.3
#3 Pandemic	453	76,175	168.2	2.4
#2 Dominion	591	64,190	108.6	2.4
#1 Ticket to Ride	114	56,289	493.8	1.9

So what’s the takeaway?  If you want your game to be wildly successful, you have to get the rules right, in a clear, easily-understood format.  Because game award juries — and we game reviewers — take note of bad rulebooks!

SO WHAT’S THE DEAL WITH LEGACY GAMES?

As a group, legacy-style games have considerably more problems with rules threads.  Here are the stats for the 5 most popular legacy-style games:

Legacy Games	Threads	Ratings	RQI	Weight
Charterstone	744	5,609	7.5	2.8
Pandemic Legacy Season 1	986	27,161	27.5	2.8
Pandemic Legacy Season 2	340	4,907	14.4	3.2
Risk Legacy	378	6,394	16.9	2.6
Seafall	866	2,183	2.5	3.8

Legacy games are obviously tough to develop, and that explains part of this phenomenon.  Additionally, BGG users might be creating threads (as opposed to looking up other ones) to avoid spoilers.  Or, as one OG-er pointed out, getting rules wrong in a legacy game has bigger consequences, so perhaps that is driving the sheer number of rules questions.  

But, in my experience, Charterstone and Seafall both had severe rules problems.  I didn’t notice it as much in Pandemic Legacy, but I did have some rules questions in both campaigns.  (More recently, I had rules questions in The Rise of Queensdale and Ultimate Werewolf Legacy, though neither of them approached the Charterstone/Seafall level of bad.)  

And some of these scores might be artificially high.  Charterstone and Seafall both had forums on other sites where the designers and publishers were answering questions, which might mean many questions didn’t get asked on BGG.

Personally, based on my own experience, I actually think there’s just needs to be some more groundwork put in on legacy games in terms of development, playtesting in front of different groups, and thinking through the different scenarios that might come up.

DEVELOPMENT MATTERS…

One way to fix this is to (a) hire a rulebook editor, or (b) hire a game developer.  Dale Yu, the editor of this site (and a friend of mine), is a game developer who has done his most prominent work on the following games, and they all ended up doing remarkably well.  I pick on Dale because I knew he wouldn’t mind, and he has a few development projects with a decent number of ratings.  

Games Developed by Dale Yu	Threads	Ratings	RQI	Weight
Castles of Mad King Ludwig	142	17,215	121.2	2.7
Dominion	591	64,190	108.6	2.4
Dominion Intrigue	112	25,825	230.6	2.4
Suburbia	195	17,712	90.8	2.8

As mentioned above, companies like CGE that put a lot of emphasis on their rulebooks also do really well.  

In short, playtesting is important (especially with diverse audiences), so is development, and so is hiring a rulebook editor.  Not all game projects have the resources for all of those things, but there’s certainly value to getting your game in front of as many people as possible pre-release.

CONCLUSION: THE PROBLEM IS GETTING WORSE

Most aspects of game production have improved over the past decade, but rulebooks seem to be getting worse.  Among the 50 newest games in the BGG 100, the median RQI is a 41.1, but it is a slightly-improved 55.7 among the 50 older games.  But the problem gets worse the newer you get. Among the 23 games published in 2016 and later, the median RQI is a mere 24.5, well below the top 100 median.  

Writing rules is tough.  I get that more than most: I’m a lawyer by trade, so I’m part of a profession that makes its living by arguing about ambiguity/vagueness and different ways to read things.  To an extent, some rules questions are an inevitable part of human nature. But I think the hobby today finds itself at a point where many of the rules questions coming up are avoidable, and that is unfortunate.

We as game reviewers need to do a better job of calling out bad rulebooks.  But the real onus is on publishers and designers, who need to find ways to add clarity.

—–

Thoughts of Other Opinionated Gamers:

Dale Y: Thanks for mentioning some of my games here, Chris – and, yes, working on rulebooks is often the main focus of my development work.  I have definitely felt that the success of a game often rides on how the rulebook is presented and understood. People can have a great time with a game when the designer teaches it to them, and any questions or clarifications can be instantly asked and answered.  But… when it’s just you and the new game, the rulebook has to be complete, laid out in a way where information is easily found, and the information in the rulebook needs to be accurate. Examples often help, though there are times when there isn’t enough space to do this.  In the end, my goal is to provide a rulebook which clearly explains the game and provides answers to all but the most obscure corner cases that might come up in play.

I often write and then re-write rulebooks to make sure that newbies can play the game.  I often try to have two or three blind plays where I watch them open the box, read the rules to each other and then play the game.  I do not comment at all on the rules nor do I answer questions. By watching how groups understand (or not) the rulebook, I can improve on the areas where questions have arisen.  And… I’m quite proud of the work on Dominion. It’s a fairly complicated game with plenty of opportunity for questions to arise given the nature of the game – each game has at least ten Kingdom cards that somehow interact or supercede the basic ruleset.  We didn’t have a section in the rules to address single card questions, and I’m guessing that if Black Market and Throne Room didn’t exist, the RQI would easily triple in value! :)

Alan How: The analysis is certainly thought provoking and it is certainly true that rules are a large inhibitor to playing the game the way intended by the designer. I think that the index has more problems to overcome. For example, the modern classics set as well as the Spiel Des Jahres winners have huge numbers of ratings. When a game is produced any questions might arise in the first year or so post launch, so the following years there are fewer new questions to ask. So across time the number of volume rises dramatically without the number of questions (threads) moving up at the same rate. Perhaps a test of this is a monthly sampling of a range of new games to see if this theory is true. The analysis of newer games compared to some slightly older tends to support this theory, I believe.

Secondly, the complexity of a game, which I am using rather than weight, will naturally favour easier games with fewer moving parts when using the index. Kingdom Death Monster has many unusual situations that develop so I suspect the vast combination of these will generate more rules questions as more expansions are added. But I’d also agree that complex games don’t need to generate many rules questions. Part of the answer in CGE’s case is that Paul Grogan is a rules editor and used by CGE as well as rules walkthroughs that Paul produces for their games. More people are using these as a way of learning the game and so reducing questions that many be left on BGG. Other companies including Portal Games have direct links on their game boxes to rules walkthroughs and explanations. It might be worth tracking these types of game as well to see if there is a correlation between games with rules walkthroughs, complexity and reduced rules questions.

I have always believed that the rules presentation is as important as the rules words. I have a personal favourite in Russian Railroads, which uses coloured sections of the rule book to distinguish different elements in the rules. It does make it easier to read and absorb information. More recently, Fantasy Flight rulebooks have a separate glossary booklet that helps with rules learning and I have appreciated their efforts to produce more helpful solutions to their games. I could be equally critical of publishers whose games fail to include helpful player aids. These can reduce rules downtime enormously but can also be imperfect when they don’t perfectly agree with the rule book.

Perhaps the best thing that this article can do is point out the need for improvement in rule writing, editing and component support. Even better point out rules that are well written and clear, and those areas where rules can be improved without spending excess time pre publication by getting more people to look at rules before they are finalized.

Patrick Brennan: Re the observation about FFG having “terrible” rulebooks, having worked with them on various LCG rule sets, I know FFG design these rulebooks with first play and readability in mind, to get the reader up and going and understanding the bulk of a game as quickly as possible, and deliberately removing any technical or side-edge stuff which would bog that process down. The aim is to make the rule-set and therefore the game as  approachable and as marketable as possible. FFG rulebooks in the main do an excellent job at this, and meet their company’s aim. They quite happily concede and accept that there’ll be a need for FAQ, and occasionally for technical documents to supplement the game. The FAQ can even be begun at the same time as the rulebook to explain technicalities that they don’t want to explain in the main rule set. (It’s pretty darn tricky writing a rule book to cover 2000+ cards when only the first 300 or so have been designed!) Now, I’m not saying this is right or good, just explaining, because this “marketing” approach to rules obviously works given FFG’s success, and illustrates that there are many aspects to be considered in determining what’s a “good” rulebook.

Jeff Lingwall: Thanks to Chris for compiling this great look at rules! Within the limitations of the available data, this is a fascinating look at the issue. With all the caveats Chris and the other commentators noted, the results do accord with my lived experience with the games in question. (I love Twilight Imperium, but I’m glad the internet exists, otherwise I might still be playing my first game?) A well-written rulebook is almost poetic: expressing great complexity in short form, hopefully beautifully done. This is not an easy task, so the designers, developers, and publishers that go out of their way to produce beautiful, useful, comprehensive rulebooks should be recognized. Perhaps we need a game award for highest RQI?

In the interactive graphic, it’s interesting to contrast games along the x-axis (same weight, different RQI): take Concordia versus Descent, second edition. They have very similar weight, but differ dramatically in RQI, which makes me think both of the companies involved, per Patrick’s comment above, but also the style of game. Or, contrast games along the y-axis (same RQI, different weight). Concordia versus Crokinole, for instance. That Concordia generates roughly the same number of rules threads per rater that a dexterity game explainable in a minute does is impressive.