November 15, 2013 Technology No Comments

Overview

Have you had a favourite Kickstarter project?

A project you desperately wanted to succeed. But throughout the campaign nothing could give you an accurate estimate of what would happen. You ended up looking at the pledges per day, which of course went down, and soon it looked like you wouldn’t reach your goal. You were on a roller coaster ride from then until the last day when that final spike of pledges managed to get you over the line.

That’s the problem I’ve tried to solve with Kickspy trends. The idea is to give you a more accurate estimate of the final outcome of a project. That way you can better gauge how the project is going and adjust if necessary.

This article describes how trends work and how accurate they are. If you don’t want to read the whole article, feel free to skip straight to a section that interests you:

What exactly are Kickspy Trends?

If we were psychic then estimating project outcomes wouldn’t be a problem. Trends for every project would be a straight line from the beginning to the end of a campaign at the exact $ value that the project finishes on:

Unfortunately we’re not psychic (as far as I can tell) so we have to rely on our smarts and a little bit of maths to make our best guesses.


Sta-Tech Wallet : Slim Wallet and Universal Smartphone Tool by Ryan Kegg

So with that in mind, Kickspy trends are predictions of the final $ pledged of a project based on the outcomes of similar Kickstarter projects. The best way to interpret Kickspy trends is this:

The current trend is the result achieved by other Kickstarter projects similar to yours

How accurate are Kickspy Trends?

I’ll split this into two parts, the accuracy of Successful projects and the accuracy of Unsuccessful projects, because the accuracies of each is a little different.

Accuracy of Successful Projects

For the most part trends of successful Kickstarter projects are quite accurate. Some are almost straight lines while others make one or two adjustments before they settle into a roughly straight line.

For those of you not so mathematically inclined feel free to scan down to the bullet points, otherwise, below is a box plot of the accuracies of a number of Kickspy predictions.

The Y axis shows when I overestimate the final pledge (+X%) and when I underestimate the final pledge (-X%). So if a project finishes on $10,000 for example and I estimate that it will finish on $11,000 then the accuracy for that prediction is +10%.

The box plot itself shows all the accuracies of all the estimates of a sample of projects taken from September 2013 for each day of a normalized 30 day campaign. The boxes show the accuracies of half the projects closest to the centre (median) while the whiskers show the accuracy of the next half of projects furthest from the centre. There are also some outlying bad predictions shown by the small circles.

What this graph shows is

  • For half of all projects, predictions are between +15% overestimated and -20% underestimated. This improves to +10% overestimated and -5% underestimated in the last week of the campaign.
  • For the other half of projects, predictions are between +50% overestimated and -50% underestimated. This improves to +25% overestimated and -10% underestimated in the last week of the campaign.
  • For a handful of projects, predictions don’t work well at all.

Examples

Below are a few examples of what the different trends look like. First is an example of a trend from the more accurate half of projects.


UNITED SKATES by Dyana Winkler & Tina Brown

This one happens to be a straight line, but most projects have one or two corrections throughout their campaign, so their trends will look somewhere between these two:

The Phoenix Project – City of Titans by Missing Worlds Media

 


 The Way of the Tiger Gamebooks, New Collector’s Edition by Megara Entertainment LLC

Finally, here is an example of a trend that didn’t work well at all (i.e. the little circles on the Accuracies per Day graph). This project managed to get just over 40% of its funding in the last day.


The Bike Index: Let’s Stop Bike Theft, Together by Seth Herr

The trends here were bad because the project seriously beat the odds when compared to all similar projects on Kickstarter. It’s also a good example that you should never give up, no matter how bleak the outlook, because there’s always a chance you can beat the odds.

Accuracy of Unsuccessful Projects

The accuracy of unsuccessful projects is a little trickier to measure because of the values involved produce very large % inaccuracies. For example imagine a project that ends up raising $10 of its $1,000 goal. If the first prediction was $1,000 then the accuracy of that prediction is +9900%.

The accuracy of trends for unsuccessful projects can best be understood with some examples.

Below are two trends of unsuccessful project; most unsuccessful projects have trends that fall somewhere between these two graphs:


ETA: Enter The Arena by Promethean Games Inc.

Mutant Football League by Michael Mendheim

You’ll notice that trends for the first one to two weeks of an unsuccessful project are overly optimistic. This is because the trending algorithm doesn’t make much use of the funding curve in that period and predictions tend to align more with other successful projects (more on this below).

The accuracy of these trends a better during the second half of a project’s life. During that period most project predictions are between +50% and +150% which then lowers to +5% and +10% in the last week of the campaign. But of course there are exceptions.

How are Trends Calculated?

It took a few months to develop the current algorithm, so there are a few nuances, but the overall technique works as follows:

To calculate the prediction for Project X, I take as many factors about Project X into account (Category, Goal, Backers, Funding progress etc.). I then compare Project X to previous projects on Kickstarter and rate how similar they are to Project X.

That gives me a list of reference projects with a score of exactly how similar they are to Project X. I then combine the outcomes of all these reference projects based on their similarity score, and that combined result becomes the predicted outcome for Project X.

When calculating the prediction I aim use around 100 very similar reference projects, but this varies depending on how many similar reference projects are available. Also the 100 reference projects tend to change over the funding period as some reference projects become more similar and others become less similar.

A note on using funding curves to measure similarity

I’ve found that the similarities between projects’ funding curves are one of the best measures of similarity between two projects.

Unfortunately in the first third of a project’s life it’s hard to match the funding curves accurately because all funding curves are very similar during that period. So for predictions during the first third of any project’s life I rely more on other similarities (Category, Goal, Backers etc).

Problems with Trends

There are currently two known problems with trends.

The first problem is that all unsuccessful project trends are overly optimistic in the first 10 days. The similarity calculations in the first 10 days of a project are based less on the funding curve and more on the static characteristics of the project. These calculations are less accurate at predicting success or failure of a project and giving successful and unsuccessful reference project the same weighting leads to trends where every project is either JUST successful or JUST unsuccessful and not very accurate for either. So I’ve chosen to weight successful reference projects higher at this stage which naturally leads to better predictions for successful projects and overly optimistic predictions for unsuccessful projects.

The second problem is that for overwhelmingly successful projects there aren’t enough similar reference projects. In that case the trending algorithm falls back to using the next most similar reference projects which tend not to make good candidates for predictions. The result is that trends for overly successful projects track very closely to the funding curve. Two examples of this are:

Trend calculation example

THE LONG DARK, a first-person post-disaster survival sim by Hinterland

The best way to understand the Kickspy trending algorithm is to see it in action. So let’s walk through the predictions for a project called “The Long Dark” with a Goal of $200,000 and a final outcome of $256,218. We’ll try to predict this outcome during the campaign.

Reference Projects

First we need a set of reference projects that are similar to “The Long Dark”. We’ll use the outcome of these reference projects to predict the outcome of “The Long Dark”. In a real prediction we would select 100 reference projects which would change throughout the life of “The Long Dark”, but for this example we’ll select the following 3 reference projects:

The Algorithm

So now we have 3 reference projects and also 3 guesses for the outcome of “The Long Dark”

But how should we combine them?

We’ll use a set of measures for how similar “The Long Dark” is to each of these reference projects. Then we’ll use those similarities to determine just how much we should rely on each reference project’s outcome for our prediction.

Again in a real prediction we would use many different measures of similarity, but for this example I’ll focus on only 2, the Goal and the Funding Progress of each project.

The Algorithm: Days 1-10

The funding progress of all reference projects are too close at this stage to rely heavily on them. As an alternative we’ll use other measures such as Goal to determine how similar “The Long Dark” is to each reference project.

  • $200K Goal makes Among the Sleep the most similar
  • $215K Goal makes Tug the next most similar
  • $139K Goal makes Pier Solar not very similar

So now let’s rank each reference project based on similarity and then give them a weighting (which adds up to 1).

  • Among the Sleep is 0.4 similar
  • Tug is 0.38 similar
  • Pier Solar is 0.22 similar

And now let’s combine our guesses based on our similarity scores

Prediction = 0.4 * $248,359 + 0.38 * $293,184 + 0.22 * $231,371

Prediction = $261,655

The Algorithm: Days 11-30

This is where it gets interesting because we can start relying on the funding curves of all reference projects.

Below you’ll see the funding progress of The Long Dark (in red) graphed against the funding progress of each reference project (in blue).


The Long Dark vs Among the Sleep


The Long Dark vs Tug


The Long Dark vs Pier Solar

During this phase, we’ll rely more on how closely the blue lines track the red line to work out how similar our reference projects are to The Long Dark.

The Algorithm: Day 11

Looking at the funding progress graphs for day 11

  • Tug is the most similar so we’ll give it a weight of 0.4
  • Pier Solar is also very similar so we’ll give it a weight of 0.4
  • Among the Sleep is not very similar so we’ll give it a weight of 0.2

And now let’s combine our guesses

Prediction = 0.4 * $293,184 + 0.4 * $231,371 + 0.2 * $248,359
Prediction = $259,493

The Algorithm: Day 29

Looking at the funding progress graphs for day 29

  • Among the Sleep was not similar but is now tracking very closely. So we’ll give it a weight of 0.5
  • Tug is very similar overall but has started to diverge so we’ll give it a weight of 0.25
  • Pier Solar is very similar overall but has started to diverge so we’ll give it a weight of 0.25

And now let’s combine our guesses

Prediction = 0.5 * $248,359 + 0.25 * $293,184 + 0.25 * $231,371
Prediction = $255,318

The Algorithm: A final word

As you can see the trending algorithm constantly re-evaluates how similar “The Long Dark” is to all reference projects and then uses those similarities to calculate the final outcome.

In a real prediction scenario the reference projects would also change because “The Long Dark” would constantly be compared to all other Kickstarter projects in order to pick the best reference projects. But hopefully this small example helps demystify the logic behind how these trends are calculated.

Disclaimer

If you’ve read this far then the following will be obvious. But I wanted to point it out explicitly:

The trends / predictions calculated for any project are just approximations of the outcome, no matter how accurate or inaccurate they are, they are not a guarantee of success or failure. Instead they should be considered as a guide only.

 

Final Word

Thanks for reading this far and I hope this gives you a better understanding of how Trends are calculated. If you have any questions or feedback I’d love to hear them in the comments section below.

Going forward I’m constantly tweaking the trend algorithm. I know it’ll never be perfect or work for every project, but my goal is to make trends as close to perfect as possible

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>