By: Chris Wray and Jeff Lingwall
Game night begins, and everybody stands around waiting for somebody to nominate a game. These days, most large groups have at least one Kickstarter enthusiast, and he or she always seems to speak first. You probably know the speech, which sounds something like this:
“I backed this game last year, and my copy just arrived. I really want to play it. It came with painted miniatures, plus it hit all of its stretch goals. My copy is a Kickstarter-exclusive!”
The enthusiasm is understandable. The backer has paid a small fortune for the game and waited a year for its arrival. Meanwhile, the designer/publisher has been sending weekly or monthly updates, causing the backer to develop somewhat of an emotional attachment to the game.
But not all of us are Kickstarter enthusiasts. In fact, among experienced gamers and game critics, most of us seem to be Kickstarter skeptics. We’re suspicious of the word “backer” — just call yourself a “buyer” — and we think the phrase “stretch goals” should be used primarily in yoga studios.
We’re not being curmudgeons: we’ve been around the block a time or two, and we know just how bad Kickstarter games can be. (Really bad.) Our time to play games is limited, so please pardon us if we’re not eager to play the game that you paid a fortune for — and waited a year for — without ever playing.
The fact that the game was on Kickstarter doesn’t mean it is a better game: if anything, it probably means it isn’t. Traditional publishers are highly selective, and they often spend years developing games, two gatekeeping functions that aren’t present with Kickstarters.
The BoardGameGeek (“BGG”) data supports us. In recent years, Kickstarter games have, on average, underperformed their traditionally-published counterparts in the BGG ratings. And as time passes, the underperformance gets worse. As we’ve previously explored, the ratings of most games decline over time. But that effect is particularly pronounced among Kickstarter games, which have ratings that decline even faster than those of traditionally-published games.
This is our empirical analysis of Kickstarter games—our attempt at partially explaining why they underperform their traditionally-published peers over time.
The History of Our Research
Nearly two years ago, we took an empirical look at the cult-of-the-new, asking if board games are getting better. We concluded, “Great games are like literature, and while every few years a book might need a new cover, what’s inside is timeless. The next time you go game shopping, check the thrift store as well as the convention floor, because in the end, a top game from 1994 or 2004 is still much higher rated than a below-average game from 2014.”
The article proved controversial, spawning one of the longest comment threads in the history of the Opinionated Gamers. We loved the debate, and it inspired us to dive even deeper into the BGG data.
Before long, we noticed a trend: Kickstarter games generally underperform their traditionally-published counterparts in the BGG ratings. We thought about writing an article on the subject, but we ultimately abandoned the project. Before we stopped writing, however, Chris sent the data to the Opinionated Gamers group, noting that the ratings difference in Kickstarter versus traditional games varied depending on the year.
The group raised several fascinating points, but three proved memorable:
- People are more likely to buy a game without having played it off Kickstarter than via retail.
- Kickstarter tends to be more of an emotional experience than just buying at retail. As Larry Levy noted, “The whole point of the Kickstarter process is to ‘grab’ the potential purchaser, via a video or some other presentation, and get them to pledge money for an unknown creation.” Plus, if the campaign is successful, backers are continuously updated about the status and production of the game, tying them even more closely to its creation.
- Kickstarting games tends to be a phenomenon for less experienced gamers. For one reason or another, many people who have been in this hobby for an extended period of time shy away from Kickstarter.
Despite the theme of this article, we’re not saying all Kickstarter games are bad…
Both of us have Kickstarter games we love. For example, we both love City of Iron and the new edition of Santorini. We’ve both “backed” many Kickstarter projects. And we both enthusiastically await releases from some publishers that use Kickstarter, everybody from Red Raven to Stonemaier. And it is not just us: some of the highest rated games on BGG are Kickstarter games.
But we’re skeptical of Kickstarter games generally. We wanted to capture the average experience with the average Kickstarter game, not the success stories (or even the failures).
We’re painting with a broad brush, and in keeping with that broad brush, we’ve made no attempt to break out games that were truly created because of Kickstarter and those where Kickstarter was merely used as a preorder vehicle. Some publishers — for example, Queen, Bezier, Indie Cards & Boards, Stonemaier, etc. — use or have used Kickstarter more as a pre-order system. They still send their games through an extensive development and playtesting period (or at least most of them do). While we didn’t crunch the numbers, we suspect many of those games would look more like traditionally-published games ratings wise. They’re still in the sample, but inclusion of those games is likely making Kickstarter games appear better than they actually are, based on our detailed review of the data.
In other words, when we say we’re not eager to play a Kickstarter game, we mean a generic one pulled of a shelf at random. We can be persuaded, be it by the publisher’s reputation, BGG averages, etc. to play a game from Kickstarter.
What causes the underperformance?
We feel the big reason Kickstarter games underperform their traditionally-published counterparts is clear: either (a) they are underdeveloped, or (b) they just weren’t that good of a game idea to begin with. Kickstarter meant game publishing was democratized, and much of the appeal of Kickstarter comes from the fact that anybody with a game idea can try to get a game published. The problem is that it may or may not be a good idea.
That would explain why they do worse. But why, then, do Kickstarter games perform progressively worse over time? In the most recent year available, Kickstarter and non-Kickstarter games have roughly equivalent ratings, and then they begin to diverge. We suspect it is due to an “endowment effect.”
What is the endowment effect? Well, it is a hypothesis in psychology, economics, and legal studies that that people ascribe more value to things merely because they own them. For example, if you gave somebody a painting and asked them the price they’d sell it for, the number would, for the average person, be higher than the number at which they’d buy the same painting. The endowment effect has been described as divestiture aversion, but the concept can also be thought of more broadly than that, since it has also been shown that people attach additional value regardless of whether divestiture is in question.
In the board game context this means people tend to attach more value to the games they own, or will soon own. The risk of an endowment effect in Kickstarter games seems particularly pronounced, because buying a Kickstarter often emotionally invests the “backer” deeply in the success of the unplayed game. This infatuation, as it were, extends into a honeymoon period as the game is delivered. But as with any relationship, the heart palpitations upon seeing the new game eventually fade, and for the game to have staying power there’s got to be more than just chrome.
In sum, if there is an endowment effect, we’d see the ratings fall over time as more and more non-backers played the game and as the backers’ honeymoon period with the game ends. So we crunched the numbers…
Gathering the Data
Last spring, we pulled the average BGG rating (the true average, not the Bayesian average) for all games (and no expansions) with 30 or more ratings published in the last five years (2012-2016). Why 30 ratings? We wanted to exclude small print runs from the analysis, plus 30 ratings is also the point at which a game gets its BGG ranking. And why that time period? Kickstarter started in 2009, and there just aren’t that many Kickstarter games before 2012. To find which games are Kickstarter titles, we pulled a list of games in BGG’s Kickstarter “family,” regardless of what was in it.
Then, we waited a year and did it again. (It turns out writing an article about Kickstarter requires almost as much patience as backing a game on Kickstarter.) This would let us compare how people rated Kickstarter and traditional games as time passed. The we started crunching the numbers. We calculated the average rating change between 2016 and 2017 for Kickstarter and traditional games, and we did this separately for games released in 2015, games released in 2014, and so on.
Is this method perfect? No, but practically speaking, it is the best data we have.
The Findings: Kickstarter Games Fall More in the BGG Ratings
We were surprised at the strong pattern that emerged from the results. In total, Kickstarter games and traditionally published games have roughly equivalent ratings when they’re released (and Kickstarter might even be ahead), but over time, Kickstarter games tend to fall faster, and in the long run they underperform traditionally-published games by 0.05 to 0.15 points. That may not seem like much, but given how small the standard deviation is for BGG ratings in our sample (roughly 0.77 in recent years from an average rating of 6.67), that’s actually a decent amount. Additionally, the size of the effect is hidden because (a) many BGG users tend not to go back and change their ratings, (b) Kickstarter games have much more self-selection bias than the more widely-available traditionally-published games, and (c) the Kickstarter sample is benefiting from us including everything in the BGG Kickstarter “family,” which actually includes a few games from traditional publishers that went through their normal screening/development process.
The graph below shows the results for games released in 2015. The x-axis (the horizontal axis, on the bottom) shows the change in the number of raters between 2016 and 2017. For instance, the outlying point on the Kickstarter side, with about 8,000 more BGG ratings, is the breakout hit Blood Rage (which we both like). The outlying point on the Traditional side is Codenames, with a jump of over 15,000 BGG ratings. The y-axis (the vertical axis, on the left side) shows the change in average ratings for the game between 2016 and 2017. For example, the average rating for Blood Rage in 2016 when we downloaded the data was 8.27, and in 2017 was 8.18. The average 2017 rating for that game was thus 0.09 less than the average 2016 rating.
Two patterns stand out. First, traditional games have an increase in the number of people rating them over time, and their ratings still declined, but less so than Kickstarters. Kickstarters, by contrast, didn’t have as much growth in the number of ratings, but their ratings change was actually worse than that of traditionally-published games.
Now, there were many Kickstarter games with lower ratings, and many Kickstarter games with higher ratings. The same for Traditional games. But on average, Kickstarter games declined in ratings by 0.13 points, and Traditional games declined in ratings by a much lower 0.09 points. (So, for example, for a popular game with 10,000 ratings, this rating difference could mean that for Traditional games about 900 people are lowering their rating a full point, while for Kickstarter games 1,300 are lowering their rating a full point.) Because Kickstarter game ratings declined by over 50% more than Traditional game ratings, we suspect there is a subtle but strong endowment effect for the Kickstarter games released in 2015.
Now, fascinatingly, this same effect occurs for games released in each prior year we have data, but by a smaller amount. In each year, the average rating decline for Kickstarter games exceeds that for Traditional games, but in each successive year the difference is smaller. The following graph shows the rating decline between 2017 and 2016, for games released in 2015 back through 2012. It’s as if in the year after a Kickstarter game is released there is a strong “cooling off” effect in which the rating adjusts down closer to its “true” unhyped rating, and in following years the game continues to cool off, but by smaller and smaller amounts.
There are several possible, related explanations. First, the original raters could be lowering their ratings over time as the game loses its shine. Because much of the Kickstarting experience is about shine or hype, the ratings drop by more than Traditional games. Many BGG users rate Kickstarter games without having played them, then the shine wears off really quickly when they do. Alternatively, later buyers — or, more likely, later players that don’t own the game — may be rating the game lower than the initial owners. It might be that later buyers didn’t experience that Kickstarter glow, or just that those who Kickstarted the game were those to whom the game appealed the most. We suspect each of these happen, but the data we have only allow us to explore at a high level.
It is also worth noting that because for Kickstarter games there is both (a) a sharper fall in ratings and (b) a smaller increase in the number of ratings, there are a couple of troubling possibilities. First, the original fans of Kickstarters are lowering their ratings at a much more rapid pace than the data suggests, or second, that the non-owners playing the game give the game drastically lower ratings than the Kickstarter owners. Without breaking out individual ratings, we don’t know, but these appear to be likely scenarios.
About Those Miniatures…
We suspect that much of the love for Kickstarter games comes from how “shiny” they are, meaning how novel and well produced they are, irrespective of gameplay. (We’ll admit that are a some beautifully produced Kickstarter games!) As a proxy for the “shiny” effect, we examined the rating changes for games that were Kickstarted with and without minis. Miniatures provide a significant amount of the hype surrounding many Kickstarter games, and the use of minis, as opposed to standees or other tokens, is generally unrelated to the quality of gameplay. The next figure shows the change in ratings for games released in 2014 and 2015. In general, ratings for games with minis drop by much more than ratings for games without minis, and ratings for Kickstarter games with minis drop the most of all. In fact, for 2015 games, the average ratings decline for Kickstarter games with minis (-0.162) is nearly twice the ratings decline for Traditional games without minis (-0.085).
While there might be many explanations for, and takeaways from, these data, the lesson we draw is that gamers are discerning. Over time, quality of gameplay will always trump chrome. While we appreciate a beautiful game as much as the next gamer, chrome sometimes acts like lipstick on a pig, and Kickstarter games are particularly prone to pork.
And, once again, we expect this effect would be even more pronounced if BGG users periodically updated their ratings, which of course doesn’t always happen.
Conclusions
Okay, that was a lot of math. So what are we saying? Our anecdotal experience often makes us groan when Kickstarter games are brought to the table, and the shinier the game, the louder the groan. While there are notable exceptions, and we applaud those exceptions, we feel that traditional publishing offers significant advantages over crowdfunding and is more likely to deliver high quality, lasting games. The statistics available, while limited, bear this out. The fact that the game was on Kickstarter doesn’t mean it is a better game: if anything, it probably means it isn’t. We’ll take our favorite publishers over a random project on Kickstarter any day. So please pardon us if we’re not eager to play the game that you paid a fortune for — and waited a year for — without ever playing.
Wow! That was a major piece of work. You could have even considered splitting that article into parts (as you are wont to do periodically). Very interesting stuff !! Thank you for your hard work on this. This gives me considerable pause about do a Kickstarter, which seemed a scary and daunting endeavor in the first place. Bob Kamp, inventor of Raven, a trick-taking card game looking for a publisher.
EXCELLENT article and analysis, Chris and Jeff. I think this helps confirm why I and many others have been VERY leery of the Kickstarter method. I am very, very judicious in the Kickstarter games I back, usually only supporting those from designers or publishers whom I trust.
Excellent article, couldn’t agree more.
Umm…I think you’re missing a point: many people rate games, particularly KSes, *before* they come out, or even exist. They rate based on theme and hype, essentially, or what the publisher says will happen. Then the game actually happens, almost no one rerates, but the actual experience is rarely as good as all the pre arrival hype. Games which can increase their rating are either extraordinary, or blew their hype attempts.
That’s a good point, and I don’t think we did miss it: “Many BGG users rate Kickstarter games without having played them, then the shine wears off really quickly when they do.”
Does bgg api expose the date rated? I would love to see the numbers on pre-release-rated games.
Looks like you used Tableau to do the analysis. Great stuff. Thanks.
Interesting article, but way far too few visuals for me – also what’s with the makeshift cirlce-box-plots? I think the authors could really benefit from R on this one ( as an example https://www.kaggle.com/gabrio/board-games-dataset-exploratory-analysis )
Very cool! I do academic research on boardgames (I see Melissa Rogerson is on your list of contributors), so this was a fun read. I’m not sure about your backgrounds (but expect it may involve data, given your discussion of SDs and Tableau visualizations), but if you’re ever interested in some more statistical support, feel free to reach out. Although if this is the same Jeff Lingwall who does “Bayesian air pollution modeling,” you probably don’t need any further assistance!
The Bayesian air pollution modeling was a long time ago, in a galaxy far, far away, but yes!
Chris, what kind of statistical tests are you using to see if these differences are actually statistically meaningful? Maybe I missed it, but I didn’t see that detail in the article anywhere. I do research for my day job, so it’s a professional curiosity. :-D
We didn’t go down the path of doing inference– that was beyond our scope here. If we went to that point we’d need to go into much more detail in the underlying data first, things we deliberately hand-waived over here (such as established publishers running kickstarters, second editions, etc).
Thanks for the reply. I guess my concern is that the article makes a claim (” The statistics available, while limited, bear this out. The fact that the game was on Kickstarter doesn’t mean it is a better game: if anything, it probably means it isn’t.”), but the claim is only *supported* if the differences discussed have statistical significance. I mean, it’s a pretty big deal. If data are being used to support a conclusion, then, IMO, we should expect some rigor when it comes to how those data are analyzed. I truly appreciate the effort to involve data — that alone is commendable — I’d just like to see some indication that the conclusions are supported by the data instead of the data being used to support an a priori conclusion. Thanks again.
Hear hear. My thoughts exactly.
But then you already know all that. ;-)
Perhaps it would have been better to say “the summary statistics bear this out.” A technical audience would then recognize that we’re not making inferential claims?
I disagree. Even saying that “the summary statistics bear this out” is a misleading claim, because as far as i can tell you get two means and compare them and say that they are different, but this direct comparison has no meaning.
You dont even state what variance the samples have, which in some cases seems to be enormous by visual inspection, and that goes a long way towards being able to establish if this data DOES in fact support your conclusion.
This reminds me of a classic example in recent history where the NHS unveiled an “amazing tool” for comparing cancer rates across the UK, so people could see where the biggest problem areas were and adjust policy. Everyone thought this was fantastic until you sorted by population size, and determined the expected variance for those samples, which clearly demonstrated that there was no regional variation in cancer rates outside of statistical expectation (except in glasgow, which is apparently where health goes to die).
I’m an American, but I’ve seen pictures of a “typical” Glasgow fried sampler platter. Frankly amazing that people can survive those!
For those who are unaware, search Glasgow Salad or Glasgow munchy box
I agree. I guess there’s an assumption the data is normally distributed, but never is there a mention of the p-value used to determine statistical difference. As a fellow scientist, I would like to see what kind of statistical analysis was done to reach these conclusions.
It’s kind of ironic that Raiders of the North Sea, a game originally funded by Kickstarter I believe, was just yesterday nominated for Kennerspiel des Jahres!
I agree with a lot of this article, Kickstarter has produced a lot of real stinkers. (It’s nice to see some numbers, even if we already knew that.) BUT, I’m just as likely to groan if you ask me to play another worker-placement, resource-trading game with a generic theme and terrible art!
After 20 years in the hobby, I’m a bit bored with traditional releases. I’m sick of the same themes, same resource trading games. There are so many new games that if it doesn’t have excellent art, graphic design and components… it’s hard to take a second look. Theme, and chrome, are big factors for me now—they really set a game apart. And Kickstarter is a good platform for both production values and unusual themes.
So let’s not ignore what Kickstarter is really good at either. I’m not going to automatically “groan when Kickstarter games are brought to the table, and the shinier the game, the louder the groan.”
…Ok, I might still groan at the Kickstarter part… but if it’s shiny enough, I’ll try it. ;)
“After 20 years in the hobby, I’m a bit bored with traditional releases. I’m sick of the same themes, same resource trading games.” That’s a good point.
Kickstarter has definitely expanded the possibility of unusual themes, which _sometimes_ end up having a great game attached. I’m not sure Chris would agree, I’m more of a thematic gamer than Chris generally (you’d see if you looked at our BGG ratings).
Hmmm….Raiders of the North Sea appears to be yet another worker placement, resource trading game with a generic (vikings again) theme?
I’m not sure Kickstarter is really the place to look for innovation. Seems like a LOT of fantasy games with lots of miniatures, lots of worker placement games, lots of deckbuilders, etc. Many of the traditional publishers are the ones who are looking for something new. If a publisher has one worker placement game in its catalogue, chances are it won’t seek out similar prototypes.
Just to defend Raiders of the North Sea, it has a really nice aspect of WP. You place a worker, then remove one. So where you go is blocked until your next turn, but it creates a nice dynamic system.
I don’t mean to take anything away from that game. Just trying to show that kickstarter games have the same problem of rehashing themes and mechanics. I’m sure RotNS is a decent game, otherwise it would not have been nominated for the Kennerspiel des Jahres yesterday!
“Hmmm….Raiders of the North Sea appears to be yet another worker placement, resource trading game with a generic (vikings again) theme?”
Fair point… but perhaps that tells us more about what the traditional industry regards as a “good game” more than what kickstarter can produce?
(And actually Vikings wasn’t a very common theme until recently. You had like Fire & Axe, Eketorp and Vikings as the main examples pre-2015.)
Still, I think you have a great point that kickstarter isn’t necessarily overflowing with innovation either, and traditional publishers are just as hungry for “something new”. I would guess though, that’s most true when it comes to game mechanisms, but maybe when it comes to themes, that publishers are a little more conservative, and kickstarter might have a bit of an edge. Just a guess.
I’ve reviewed my game purchases since 2015 and I was very surprised to see more than half are from kickstarter! There are a few big box editions (Carson City, Tokaido, Shogun) but there are also a lot of solid titles: Tuscany, Scythe, Trickerion, Anachrony, Defenders of the Last Stand, Ancient Terrible Things, Burgle Brothers, Blood Rage, Space Cadets: Away Mission, Spurs, The Gallerist, Spirits of the Rice Paddy, Tavarua, Mare Nostrum… and 1 or 2 stinkers I won’t list. :)
The article puts a significant amount of stock in BGG ratings being reflective of the likes of those who back KS games, especially ones with minis. As a long-time gamer who straddles both board gaming and mini gaming, my anecdotal experience is that mini-gamers will back KS games with minis, and most don’t interact much with BGG. If they did, then Warhammer 40,000 would have many, many more ratings.
It’s also a bit disingenuous to say KS games aren’t subject to the same rigour as traditionally published games, then identify ones which are, such as Scythe or Blood Rage, and say they’re not really KS games and thus are skewing the figures higher than they should be. In doing so you’ve created your own definition of what KS game is and are passing judgement based on that.
Those quibbles aside, a very enlightening piece of research :)
We don’t have data on this, and I’m not sure it exists short of handcoding the development process behind a thousand games, but my anecdotal evidence would say games on KS _in general_ don’t have the same level of development. There are happy exceptions like Blood Rage– which we did want to point out! For instance, I Kickstarted the second printing of Gloomhaven and am terribly excited for it. It sounds like a well-developed game, and thematic Euros hybrids are generally my highest rated games.
That may well be true. It certainly seems like a reasonable enough assumption. Though I was specifically referring to things like –
” They’re still in the sample, but inclusion of those games is likely making Kickstarter games appear better than they actually are, based on our detailed review of the data. In other words, when we say we’re not eager to play a Kickstarter game, we mean a generic one pulled of a shelf at random. We can be persuaded, be it by the publisher’s reputation, BGG averages, etc. to play a game from Kickstarter.”
Those games are still KS games, and they don’t make KS games appear better than they are because they *are* KS games. By raising this you create an alternate definition of what a KS game is.
Since posting the previous comment I came across a former member of the Malifaux design team make much the same point about the minis market. That is a very successful and very well received game, but it has virtually no presence on BGG because it appeals to a different audience. It is worth bearing this in mind when analysing what BGG ratings tell us about KS minis board games.
Again, though, I do wish to reiterate that it is a very impressive piece of research which many points of useful data, and I don’t wish my discussion of these points to suggest I think otherwise :)
You’re kind, and make good points!
sounds like a “no true Scotsman” fallacy where any well-designed, play-tested game from an established publisher is labeled “not a real Kickstarter” and is accused of/blamed for “skewing the data”. This points to the confirmation bias of the authors who set out to prove that Kickstarters aren’t as good and then disregard any games that don’t support that. The “average” Kickstarter has to be averaged not just from the failures but from the successes, otherwise, it’s not the “average”.
What we_really_ ought to do is run a Kickstarter to raise funds to do an experiment to probe the effect of Kickstarter! :-)
First, to be clear, we did categorically count as a Kickstarter anything that was in that BGG family. Second, I’d encourage you to spend a few minutes browsing through the BGG Kickstarter “family” to see what we are saying. You’d notice that there are a number of games in there that existed in very successful form before Kickstarter, and I for one don’t think it’d be proper to truly classify them as Kickstarter games. For example, Circus Flohcati and Metro. Additionally, there are countless more games that were in final form and Kickstarter was merely used as a preorder system. I think it is worth making a distinction between games that exist solely because of Kickstarter and games that already existed but Kickstarter was a convenient preorder system.
the problem with making such a distinction is it doesn’t matter in terms of the narrative you based your initial assessment on (“we groan when you want to play your Kickstarter game the finally arrived”) and the effects you discuss, including the endowment effect will apply to those games as well. For purposes of the analysis you are making, there is no difference and it’s disingenuous to draw a conclusion about “average” kickstarters while dismissing the upper end (even if that upper end was included in your data)
I agree with your point about the narrative: that’s in part why we left them in. (Also, it would have taken a lot of time and individual judgment to take them out.) And the endowment effect would absolutely apply to those games as well. But I do think it is worth noting that the many of the most successful Kickstarters have been from very reputable publishers on games that already existed or would have existed regardless of crowd funding. Remove those, and the data looks even worse for crowdfunded games.
I, too, think it is important to differentiate between a KS game (first released through KS) and one that has already been published and KS is being used as a preorder system for the new edition of an already-published game. The former is usually less developed and tested, while the latter has already sold out once, and many of the demand comes from people who played the first edition.
I agree that a game which pre-existed as a traditionally published game and was re-released via KS can reasonably be counted as a traditionally published game, but the KS version of it is a KS game also. To take a recent example, it’s unlikely Brass would have got a high production value reprint without KS, therefore the high production value reprint of Brass is a KS game, even if it also a trad published game. There are interesting things which can be said about the appeal of high production value games and KS just from Brass alone.
Games which are the proverbial ‘pre-order’ are definitely KS games. If you don’t count them, or rather if you do but say they skew the figures higher than they should be, then you have definitely placed your own defintion of what a KS game is, and it isn’t simply a game that raised funds on KS (or other crowdfunding platform).
That the games are further judged by BGG ratings alone means two significant biases are introduced into the results, and these are not as well accounted for as they should be when you draw your conclusions. The research still has much merit and can say many interesting things. That people are more likely to buy a game they haven’t played via KS than retail is probably a safe assumption to begin with, but the data helps bear that out. Ditto with regards to production values being a very important part of the appeal of KS. You need to be very careful though when saying that the data proves KS funded games are not as good as trad published games. It is safer to say that the BGG community prefers trad published games, and that they prefer KS games which could conceivably have gone down, or have previously been down, the trad publishing route.
I wrote “preorder of a previously published game”, not preorder of an unpublished game.
Basically, with BGG ratings, you need to make a distinction between games previously published (that are known) and games that do not exist without a KS campaign. Brass exists (if only on the second-hand market, now), whether it gets a new edition or not. It’s still not going to be a perfect science, as some first print runs are very small, and what do you do with something previously released as a free Print-n-Play? It gives me a headache just trying to think about it, which I why I didn’t contribute to this article :-)
Nicely done analysis! I’d seriously consider pitching FiveThirtyEight on this set of data (they’re not just politics and sports).
I can’t help but wonder if experienced gamers rate higher or lower than newer gamers, and whether the pattern is that of ‘experienced gamers lean higher on traditional games, newer gamers lean higher on Kickstarter’…
Thanks for the kind words and critical thoughts, everybody. We’ve tried to reply to everything, and it is taking a fair effort, so we might slow down. And we’ll hopefully have some follow up articles based on feedback.
I will say, though not everybody agrees with our conclusions (which we knew would be the case) and opinions, this has been a great discussion around the internet. The gaming community really is awesome.
I echo Chris’s comments. One comment I’ve seen here and Facebook is people wanting a more sophisticated analysis– after all, we did use the word “statistics!” No, we didn’t do a regression analysis, and heavens no we didn’t cluster our standard errors by year of release, or check for normality, or run a Bayesian hierarchical model, or look around for credible instruments! We didn’t even consider doing those things not because we’re lazy, but because this isn’t an academic article …. We were pretty open in the article about the limitations in the data, and believe that despite those limitations there seemed to be a story to tell based on some simple graphs and summary statistics. That’s all.
As Chris says, great discussion, and great community.
Impossible to measure, but I would just about guarantee that there is a certain amount of “downvotes” on KS games, purely because they are KS games, and having nothing to do with their merits at all. I suspect that there are even people going out and down-rating games that they have never played, based entirely on the fact that they were pitched via KS. And that’s going to skew the numbers, ultimately.
This is really interesting, and I love the number crunching approach to evaluating the relative quality of games published on Kickstarter vs. traditional publishing. However, I can’t help but think, given the fractional differences being discussed, whether it’s a bit more of a “tempest in a teapot” situation. While I don’t want to minimize the trend that evident, it’s still the case that the relative difference between a game rated 7.4 and 7.25 is pretty small to the average gamer.
Which brings up another point. When it comes to the nominal scores involved here, all the article states is that “Kickstarter games and traditionally published games have roughly equivalent ratings when they’re released (and Kickstarter might even be ahead).” That statement bears further examination, because there is no discussion (although maybe it will be in another article), of what the actual ratings are for KS Games compared to “traditional” board games. It would be extremely relevant to know the starting points from which these declines occur. So, for example, a “huge” decline from 8.8 to 8.6 has a much different meaning than a “small” decline from 6.09 to 6.06. Putting aside the fact that a game rated an 8 is likely much better than a game rated a 6, one would expect that the decline would be larger if the starting median score is so much higher to begin with.
And if the decline over time results in KS games going from being rated fractionally higher that traditionally published game to being rated fractionally lower, then is that really a meaningful difference, even though the decline was greater for KS Games?
All of this is to say that this article is really interesting, but there is so much more analysis that I’d love to see to give a better context to what the data means.
Curious how GMT’s P500 is viewed… as a Kickstarter of sorts, or not… Twilight Struggle was “crowd funded” to an extent, right? I could argue that 9 of the top 100 BGG games are crowd funded efforts. Certainly does seem that a few have risen to the occasion, but you might be generally correct that crowd funded games don’t reach the heights of traditionally published games.
Interesting analysis but my experiences are completely different. Most of my favorite games are kickstarters. For example Fallen, Scythe, Zombicide, Gloomhaven … It seems to me, the traditional big boardgamecompanies don´t want to take the risk to give a revolutionary new gamemechanic a chance. They always try to make their games compatible to the mass market. In mechanics and themes. Especially here in germany. And this is why kickstarter shines – at least for me.
Thanks for the detailed article. It’s interesting to see this data being quantified. There might indeed be some trend behind the data due to a) hype b) Kickstarter ‘filtering’ people to those who are the most interested and enthusiastic in the game.
We saw people giving a rating of 10 to our game Saltlands before even receiving it. We said ‘what?’ – so yeah, it’s understandable.
The problem is that your analysis as you have published it is not a really a statistical analysis that could be used to support your hypothesis of “KS games being worse than traditionally published games”.
All I see is a bunch of circles on top of each other. Have you tried fitting a distribution on them and calculating the standard deviation and the expected value of it? Identifying outliers? Compensating for the small sample amount in some cases?
I’m not saying that your hypothesis is wrong, because we all generally agree on it, I’m just saying that it’s hard to statistically prove any hypothesis by drawing circles on top of each other, because our intuition, our eyes can be easily misled by the presentation of statistical data. (We all know Churchill’s stance on statistics ;))
Or if you calculated these, then you could publish it in a next blog post… ;)
Thanks!
I’m missing some data, maybe I just didn’t see it?
What was the average score for each type of game in the list? I also didn’t see how many games were included in each table. These seem important – I mean, if KS games dropped by 0.13 from 8.5 to 8.37 and reg game dropped from 7.5 to 7.37, than the conclusion doesn’t really match the data?
Last I looked, there were twice the number of games tagged as KS in the top 1,000 games than the norm; while that isn’t as deep an analysis as yours, it does suggest an opposite trend than the one you saw, which makes me skeptic about the entire thing.
I don’t believe that the issue that you have is with crowdfunded games. I believe that the issue that you have is with games that are under developed. This is likely more a correlate of publisher size and experience than it is of funding source (though an indirect correlate with crowdfunding is likely as more inexperienced/small publishers go the crowdfunding route).
So what you need to do in your analysis is compare like for like. Compare “established” publishers that use crowdfunding against equivalent size/experience publishers that use traditional publishing models. You should also compare publishers of different sizes (in numbers of published titles) and do so both on and off crowdfunding models. Then, I believe, we will have a more meaningful analysis.
Developed vs Under Developed is a great point.
I’m not sure publisher size is the best way to judge that, but it is another interesting angle.
Good point, and I think that is the crux of the issue. Think about this: a traditionally published game has to sell AFTER it’s produced and sitting on store shelves. It darn well better be well developed, otherwise it will stay on those shelves. And the games are available to play before I have to purchase a copy (another gamer or game group usually snags anything that looks promising).
I always thought that a Kickstarter game is usually funded by those who have not played the game, and backers all get the game at the same time, so there’s no opportunity to play the game before getting your own copy. Publishers need to focus more on the “elevator pitch” to backers, as well as the chrome to get people excited about it.
Great article guys. Having touched a large number of KS games in the past 9 months with Man Vs Meeple, I can attest that it’s not always a pleasant experience to work on KS titles (though the same holds true for traditionally developed and delivered games as well). I try to walk both lines of this debate for obvious reasons and always attempt to stay as neutral as possible when discussing KS vs Distribution. However, having just looked through our list of KS videos since MvM was created, I can most definitely support the data above – and I honestly feel like the market is shifting. Games with incredible art and miniatures are now barely funding, where even a year ago these games would have hit 200k+ in hours after being released on the platform. I feel like consumers have become smarter and have looked past the obvious chrome to decide for themselves if these said games are truly worth backing.
KS in general has an incredibly varying degree of highs and lows, that has always been the case – yet every once in a while a non-traditional publisher will surprise everyone with something that is truly worthy of respect and admiration. It doesn’t happen often but when it does I am genuinely happy to see these particular games be produced and made available to the public.
I believe that the one main keyword to take away from this wonderful article by Chris and Jeff is the word “Developed”. I would challenge potential KS creators to look honestly at your game and to search outside your personal reach to find seasoned gamers, industry personnel or even media people like ourselves who can give you honest and sometimes even brutal critique about all aspects of your product before going to KS. You have one shot to make a first impression and if any part of your project seems under developed then the masses will quickly decide your fate.
Great article guys – way to take on a tough and sometimes touchy subject.
Jeremy
What stands out to me is that for every generality you draw, you follow it with a list of exceptions that show your conclusion is suspect. Of course the top end of the range “skews the average.” So does the bottom end. That’s what statistics are about.
In your summary you say, “We’ll take our favorite publishers over a random project on Kickstarter any day.”
Well… duh. Would you take a random non-KS game against a KS game from a favorite publisher? Or even against a random successfully funded KS game? ‘Cause let’s face it, there’s a lot of bad games that come from every source.
My suspicion, after reading your bit (and skimming the comments), is that traditional publishing is probably better at producing the very BEST games…. but that KS holds its own with the great and merely good ones.
Bottom line: If you are turning your nose up at a game simply BECAUSE it came from Kickstarter, you could be denying yourself some really fun and enjoyable games. The Kickstarter Guy in your group may have great taste in games… at least better than the guy who only wants to play Munchkin.
Pingback: A lot of research on the question of Kickstarters | The Tao of Gaming
While a good read, I think we have to remember the point is not if they are all better or if they are all worse. The point is more games are available to the marketplace and the gamers. So that is better.
Who really cares if you want to support them, all due respect. Because you, like all of us are just a statistic as well. You fall under the “those that dont support KS” on the chart. And for each of you there are many that do support. So cool story, but really who cares, there are more games to look at due to KS and that, my friend, is the point of KS. It is not a quality control, it is a market expander.
Well, there are obviously quite a few people who care, judging from the comments and (sometimes much too) passionate discussion on other social media forums :-) For me, it matters in two ways: 1) as a gamer, I, too, am more open to play a game from a traditional publisher than a Kickstarter game, and 2) as a game designer, I am more open to working with a publisher that believes in my game enough to support it with an initial print run. But if other gamers want to play Kickstarter games, or other designers want to test their games with crowdfunding instead of through established publishers, more power to them.
” But if other gamers want to play Kickstarter games, or other designers want to test their games with crowdfunding instead of through established publishers, more power to them.”
You make it sound as if every designer has the possibility to find a developer easily.
But that was exactly the point of kickstarter, and in a lot of cases still is: it’s a good way to get your game out to the public even when no publisher wants to touch it, or you have other reasons not to go that route. Which, as
What irritates me a bit about people that have a bias against kickstarter is that this bias seems to cloud their otherwise good judgement, leading them, for example, to publish articles like this (I won’t go into details of the shortcomings, they are all listed here in the comments).
I usually really like TOG, but this was rather disappointing.
But it’s not a bias if it’s based on experience, Christoph. Lots of people love KS games and it’s great that their available. But the ones who are making critical comments about them are doing so because of the many negative experiences they’ve had in playing them, considerably more so than with standard published games. It’s not like they have some deep-seated hatred of the KS process; they just don’t like the games as much.
But that’s part of the bias: Lumping all kickstarter games together because of bad experiences with some of them.
The Opinionated Gamers themselves do make distinctions: they admit they’ve enjoyed and liked games published through kickstarter, but somehow they seem to not count them as “real” ks games because they were thoroughly tested/published by established studios/designers. But to make this distinction is disingenous, as others have already pointed out: all of these games are kickstarter games – you don’t get to redefine something so it better fits your argument.
So, it’s fine if you’ve been burned by some misinformed pledges and are more cautious now – but to make some general assertions (and fail to back them up with data) is something else.
What I’m also really tired of hearing is this “The game is high on bgg because kickstarter backers are more emotionally involved” nonsense.
Sure, there might be some truth to it, but without any empirical evidence, this is just a bold claim that subliminally is more like “I don’t enjoy this game, so I’m looking for reasons people that like it are wrong”. And that’s a really childish way of thinking.
I can only speak for myself and I’ve had serious issues with the vast majority of KS titles I’ve played. To the extent that when someone offers their shiny new KS title at game night, I look for other options.
But I agree, one of the problems we have here is terminology. It isn’t KS games per se that I have problems with; it’s games that haven’t undergone professional game development. That doesn’t mean such games aren’t any good, merely that they usually don’t work for me. But that’s why these distinctions are being made; in my opinion, it’s the non-developed KS games that folks have issues with and there doesn’t seem to be any shortage of them. So all KS games are getting lumped into the same pile, which may not be fair. But most of the KS games I encounter seem to be from that group, so I think that’s why it’s happening. I don’t think it’s weasel-wording; I think it’s because those are the designs that truly are the offenders.
My impression of the article is exactly that: we don’t like playing games that are underdeveloped, and Kickstarter has a higher proportion of underdeveloped games than traditional publishes. Therefore, it’s safer for us to avoid KS games altogether at our gaming group, unless we’ve had some time to look at the rules, hear more feedback from others, etc. That doesn’t mean all KS games are automatically underdeveloped or should be avoided. Like everything else in life, it’s not black and white, but we ARE talking about board games, not people.
Before KS, some designers who could not get their games published by existing companies still self-published, but the risk was far greater, as they had to produce the games first. Some still foolishly developed their games in a vacuum of friends and family, and wound up with a garage full of unsold games and a debt from all the costs.
I personally think the IDEAL of KS is awesome, giving creators an opportunity to create without risking a mortgage on their family home. But human nature shows that when presented with shortcuts, most people will take them, and if KS game backers demonstrate that they value components over game development, then that’s where the effort will usually go.
So, can a KS game be well-developed? Absolutely, but the article (and my personal opinion) point out that those are the exceptions. Can a traditionally published game be underdeveloped? Again, absolutely, but those are, again, the exceptions, and usually those come from new self-publishers.
With most of my own published games, professional developers from my publishers have been instrumental in making the games better, and I seek out publishers for my prototypes who have a reputation for good development, even if I have to wait a few more years.
“My impression of the article is exactly that: we don’t like playing games that are underdeveloped, and Kickstarter has a higher proportion of underdeveloped games than traditional publishes. Therefore, it’s safer for us to avoid KS games altogether at our gaming group, unless we’ve had some time to look at the rules, hear more feedback from others, etc.”
1. Imo the article does assert a lot more than that. For example:
“The BoardGameGeek (“BGG”) data supports us. In recent years, Kickstarter games have, on
average, underperformed their traditionally-published counterparts in the BGG ratings. And as
time passes, the underperformance gets worse.”
That’s a bold claim, and unfortunately, the article doesn’t really provide proof of that. On the contratry, they later admit:
“In total, Kickstarter games and traditionally published games have roughly equivalent ratings
when they’re released (and Kickstarter might even be ahead), but over time, Kickstarter games
tend to fall faster, and in the long run they underperform traditionally-published games by 0.05
to 0.15 points.”
So, they contradict their own previous statement, that kickstarter games underperformed (from the start). They are right in asserting that, over time, they tend to underperform. But to me the numbers don’t look significant. And the people here that questioned the (pseudo-) scientific approach used here seem to be of the same opinion.
It also doesn’t really inspire confidence in the method when they are also making false claims (probably based on calculation errors):
“But on average, Kickstarter games declined in ratings by 0.13 points, and Traditional games declined in ratings by a much lower 0.09 points.[…] Because Kickstarter game ratings declined by over 50% more than Traditional game ratings, we suspect there is a subtle but strong endowment effect for the Kickstarter games released in 2015.”
0.09 to 0.13 is not “more than 50 percent more”, but about 45%. That might be considered nit-picking, but again, if you use data to support your argument, you should make sure to use it correctly.
But apart from the factual accuracy of the article: Wouldn’t you agree that the stance ” it’s safer for us to avoid games altogether at our gaming group, unless we’ve had some time to look at the rules, hear more feedback from others, etc.” is valid, and even necessary, for all board games, not just kickstarter ones? And if so, why make a whole article about that tautology?
It probably boils down to “Kickstarter has a higher proportion of underdeveloped games than traditional publishers”. And this is a claim that I don’t see supported by anything in the article, nor in your comments. Instead, it seems to be an (informed) guess. I understand that one can easily get the impression this is the case, when confronted with mediocre games from ks again and again.
But are you positive that the tratitional publishing model is really better at preventing underdeveloped, or just plain bad, games? I’m not so sure of that. The number of board games published each year is huge, and tratitionally published ones are still the majority, I think.
And I think there’s a not insignifican number of duds included. The thing is, you are very unlikely to ever have to play them, because word of mouth or reviews will usually prevent them from being widely circulated. And that’s where I kind of agree with the article: Kickstarter games tend to get more attention even if they are bad.
“I can only speak for myself and I’ve had serious issues with the vast majority of KS titles I’ve played. ”
I understand that your experience influences your judgement, but
a) this is supposed to be a data-driven article about objective facts.At least that’s what it sounds like. Unfortunately, it doesn’t succeed in doing so.
b) maybe your experiences are not necessarily the norm? Brent futher down assert that he’s an experienced gamer, has backed lots of games, and his hit rate is about the same as with published games. My own experiences with Kickstarter are also largely positive.
The most important thing is to inform yourself as much as possible about a game before commiting – but that is true for all board games.
And, as mentioned below, many campaigns are quite open about gameplay/components etc., more so than some traditional publishers.
If you look at the bgg ratings, 11% of the Top 100 are ks games,and more than 14% of the Top 1000. That’s not at all a bad amount.
And I do wonder whether you have played (or maybe just disliked?) games like Viticulture, Above and Below, Gloomhaven, The Manhattan Project, Trickerion, Scythe, Keyflower, Blood Rage, Arcadia Quest, any Lacerda game, Arcadia Quest, The Resistance: Avalon, Coup, Flash Point, etc. etc. Maybe you had to endure a few bad games for every good one? Or maybe you were just unlucky to play the wrong ks games?
I love Kickstarter and am a very experienced mature gamer having been in the hobby for 45+ years.
The third “memorable” statement; “Kickstarting games tends to be a phenomenon for less experienced gamers. For one reason or another, many people who have been in this hobby for an extended period of time shy away from Kickstarter.” smacks of snobbery because none of the data presented here addresses this somewhat condescending point of view. Many of my “experienced” gamer friends also back games on Kickstarter.
Two of the games in the top 10 (as of this writing) BGG games are Kickstarters – they seem to be holding their own. I would suggest that the drop in ratings that has so much emphasis here is unfair. There are lots of factors that would impact the ratings drop. The article mentions the honeymoon factor, and I would bet this is the biggest factor. Kickstarter buyers rate their purchases higher than non buying players afterwards. This is a well known phenomena, folks value something they have an investment in higher than folks who don’t have that same investment. This is the biggest reason using the drop in ratings is completely unfair and frankly biased.
I have backed around 300 tabletop games projects and have more than a few dogs to be sure. I would suggest that I have had very close to the same percentage of dogs come from traditional publishing as I have had from kickstarters. Not all publishers do the kind of in depth development that many folks think they do.
There are older Kickstarter games still hitting my game tables all the time. They just don’t happen to be the kinds of games that the average Opinionated Gamer seems to prefer, which is really what I think is at the heart of this dislike of Kickstarter by many on this website.
Love it or hate it – Kickstarter has changed the business model for game publishers and it will continue to do so. These folks; http://icopartners.com/2017/01/kickstarter-2016-deep-dive-games-category/ suggest tabletop games in 2016 made over $100m dollars. There are increasing numbers of publishers who are not going to ignore that number, so get used to it.
Peace
Pingback: Games News! 29/05/2017 | Shut Up & Sit Down
Is it possible to get access to the raw data?
I would add that using the BGG rating as some “objective” measure of the game’s quality and longevity is a very dubious assumption and starting point on which the whole analysis is build. If that was true, chess and go would certainly be perennial leaders. BGG certainly has a dominant position in the hobby as a resource database, forum, etc but the “active” users probably don’t represent the gaming scene as a whole. Case in point – most of the (male) BGG users would probably rather have a prostate exam than play Monopoly, yet the game is still being played (and liked) by millions around the world. The BGG community is IMO skewed towards heavier euro-style games (Puerto Rico is ranked very high, yet rarely played regularly nowadays). Wargamers can still use the BGG as a resource and forum, but many excellent wargames are ranked outside of the Top 1000 because they are niche product even for boardgame enthusiasts. But if you have found your wargaming buddies to play those games, who cares about the rating?
And that is the issue that separates the board games from something like books or movies. If you like some obscure Chinese indie movies or trashy novels, you can happily disregard IMDB’s ratings or Amazon rankings and still enjoy them. Unless you are a solo gamer, you will need to persuade other people to spend time with you and the game you wish to play. And that is up to you and your social circle or particular gaming group. But remember, that a groan from your gaming group when you bring Terra Mystica on the table is equivalent to your moaning that your little cousin wants to play Monopoly with you yet again.
Pingback: Kickstarter Burnout?
Pingback: ChangeMaker » Co tam w sieci piszczy #28
Pingback: Podcast #118 – En busca del juego de Oz
Pingback: Let’s Talk About Rule Quality (Article by Chris Wray) | The Opinionated Gamers
Pingback: Board to death: reasons to stick with video games - SinisterDesign.net
Pingback: Acknowledging Colleagues' Excellent Advice About Crowdfunding