HOW TO ANALYZE DATA LIKE THE UNICORNS
The startup world is booming, and every year about 150 billion dollars are invested in startups globally. Most startups hope to disrupt an industry, improve the life of their users and of course bring in some cash. Together startups, VCs, business angels, and corporates want to join the ride to build the future.
A few of the many startups have succeeded and really changed the world as we know it, we often refer to them as unicorns, because they are so very rare (not inexistent though, as real unicorns, puh!). There are today 235 startups in the world valued above the magical threshold of 1 billion dollars who are currently private and are therefore part of the unicorn club. Companies such as Uber and Airbnb are a part of this prestigious group of startups, while Spotify, Facebook, and Zalando used to be a part of it before going public.
But what have these companies done to reach success meanwhile many others have failed? Is there a secret or is it only pure luck? As earlier mentioned, startups who know more about KPIs seem to experience faster growth, but is that something that we also can recognize among these giants? When searching through the world wide web and talking to the network around me to find out how these organizations work and how they reached success, one thing became clear. Data and metrics seem to be central to these companies' operations and culture.
We all know, that culture is essential for a company. But what does the culture seem to be among the startups in the unicorn club? Being metrics driven and referring to data during the decision-making process seems to be something deeply rooted in the company culture. All the startups' team members seem to refer to data as if it was something obvious.
We can look at Spotify as an example of how new team members react to the culture when they start. Jason Palmer, Engineering Lead, was not used to working with big data before starting at Spotify and describes the culture as captivating. "The culture has a way of engulfing you in a data-driven mindset. After working at Spotify for only a few months, I was talking about term weighting and signing up for internal courses on the R programming language".
At Facebook, their data-driven culture is reflected in the size of their database. Back in 2010, when Facebook "only" had about 500 million users, they recorded 4 terabytes of data every day. That represents 3 MB of data per user per year, how much data does your startup record per user? Adam Mosseri, today VP of Product Management at Facebook confirms that the company has invested heavily in their data technology and adds that "we believe it's important".
At the core of the data-driven culture, we find experiments. It rather makes you think about scientists instead of startup geeks, but it seems to work for them. At Spotify, everything is tested and every outcome is measured, no matter if it might be a new tool, weekly meeting schedules or features. In that way, Spotify manages to avoid arguing and their decisions tend to be data-driven instead of opinion, ego or authority driven. The data-oriented culture seems to be one way to create a less political and friendlier organization.
Be like a scientist, do lots of experiments. Testing is an important part of a data-driven culture.
As growth is a part of the definition of a startup, marketing is the tool to get your rocket up to speed. Thanks to the digital marketing revolution over the last couple of years, marketing has never been so dependant on data as today. There are a large number of channels out there for your startup to try, but how do you know which ones to invest in?
If we look at Airbnb, they have discovered the best-performing marketing channels by testing most of them and performing rigorous AB-testing to identify the winners. One example of this was when they wanted to improve their number of hosts in France. Thanks to market research, Airbnb knew that there were many common vacation areas that they needed to cover to dominate the market. They tried out classic Facebook ads, but with disappointing results.
Rebecca Rosenfelt was at the time Product Manager on the Growth team and she tells us about how they proceeded "First of all, we created kind of this scrappy AB-test so we selected small vacation markets in France that we thought would be desirable for Airbnb, then we randomly selected half of them that we would physically go into and half that we wouldn't". The company then sent small teams into the chosen markets that talked to the locals, organized meetups and placed low-cost ads.
After the end of the project, the results were compared with the control markets. Thanks to Airbnb's discipline regarding data they found out that the test markets performed a 5x lower cost per acquisition. Even though this experiment only lasted for a limited time period, the test markets continued to grow 2x faster compared to the control markets that only received Facebook ads. Rebecca adds "If something is not working, get in there and figure out what's happening, you actually might be surprised that your unscalable tactics will be surprisingly scalable or at least you'll learn something about why it isn't working and what might be able to work". Today, France is Airbnb's second-biggest market that in 2015 generated a GMV of €218M.
If we stay in Europe, we find another example of data-driven marketing at Zalando, the e-commerce fashion giant, during the time they had their biggest growth spurts. Christian Meerman, is today a founding partner at Cherry Ventures but used to be the CMO of Zalando. He tells me about when they started experimenting with TV as a marketing channel, "In 2010, we set up our first TV-commercial, which of course was not an exception from our performance marketing management. We used advanced forecasting methods to attribute visitors and sales to each individual spot, making it possible to tweak factors and make the commercials a successful marketing channel".
As for the example at Zalando, data isn’t only used to choose the right marketing channels, but also to perfect them. This was also done at Uber to maximize the growth. Thanks to extensive use of online advertising to attract drivers, the company was able to exactly figure out the factors that motivate a person to sign up as a driver. Are they working part-time or are they unemployed, where do they live, do they own a car? Some argue that Uber is as well informed as the US Department of Labor about low-wage workers. That database became an important success factor for Uber to target the right audience and expand the number of drivers as quickly as possible.
Data is also playing a large role in product development among these successful startups. It helps them to focus and tweak to guarantee success since the biggest risk is building and wasting time on the wrong things.
Adam Mosseri shares how they proceed at Facebook to check updates before rolling it out to all users: "We generally sanity check our changes with data by running AB-tests, at our size we can launch a product to a small percentage of users, half a percent, and get statistically relevant data very quickly". If the test group shows a positive change, more time will be invested to make the feature ready for a full rollout, if not, the product is tweaked until showing the desired result, or the team goes on to work on alternative features.
Furthermore, Facebook do something they call waterfall analysis of their workflows, to see how many users drop off in each step. A case might be the image uploader, then the team would look at the number of people clicking the button and look at how many go through all the steps and finish the complete process. “We use data to sort of optimize a workflow, and we're very comfortable doing this" explains Adam. The waterfall analysis helps the team identify abnormalities and create hypotheses for improvement possibilities.
Indeed, the lean startup principles, as first stated by Eric Ries, seem to be followed at a large scale at many of these successful startups. At Spotify, they develop their product based on the mantra, "Think it, Build it, Ship it, Tweak it". The product development process is based on multiple steps according to the mantra.
Before starting to build a new feature, the responsible team inform themselves with research, is this something that people want? Moreover, they define a one-sentence pitch like "Radio you can save" and define the hypothesis to answer how they think the feature will impact user behavior or core metrics.
The team then builds multiple prototypes that are tried out to get a feel for the feature and study reactions. When they feel confident enough, they go out to build an MVP (Minimum Viable Product) which is then tested on a small percentage of users. Thanks to this initial test based on AB-testing techniques, the hypothesis can be proven or disproven. During this period, the team might tweak and redeploy the feature multiple times until they see the desired impact in the data. When the feature fulfills the hypothesis, it will gradually be rolled out to all users while taking the time needed to sort out practical stuff like operational issues and scaling. By the time the product or feature is fully rolled out they already know it's a success because if it isn't, they don't roll it out.
All of these tech companies have very clear goals of what they want to improve, which help all teams to focus and prioritize tasks. In the case of Facebook, the objectives were for a long period of growth and engagement. Based on those two objectives the core metrics were defined. Adam Mosseri explains how they defined their core metrics at Facebook: "Growth being defined as how many users come on to the site and engagement being defined as how often users use the site".
If we go back to the example of Airbnb, the focus is also on growth, but rather in terms of GMV (Gross Merchandise Volume). To optimize on GMV, they need to effectively acquire loyal users, which is the focus in everything that Airbnb does. Rebecca from Airbnb shares some advice when it comes to focus, "another thing that people do who are more new to data, is that they will overmeasure. It's really important to think about the really important metrics, quantity does not equal quality when it comes to data. So, for example, when we did the experiment in France, we knew that the main thing we wanted to look at was CPA (Cost Per Acquisition), so we didn't really measure anything else besides the cost and then how much listings we got from them, and that really helps in terms of simplicity".
Also at Zalando, Christian and his marketing team knew what to focus on to succeed. "At Zalando, we invested heavily in marketing and brand building, but to ensure a strong return on investment and performance we were extremely data-driven. Compared to other companies, both corporates and many startups at the time, we tracked every euro we spent and continuously looked at metrics like CLV to CAC ratio, CRR (Cost Revenue Ratio per channel and advertising partner), individual RFM scores per customer (Recency, Frequency, Monetization), and many more. This helped us to stay oriented towards growth and healthy customer acquisition metrics". Today, 8 years later, Zalando has a yearly revenue of €4,5B and over €200M in profits.
It is of course not easy for early-stage startups to copy and paste the processes of tech giants. However, they can be a very good source of inspiration and I hope there are a couple of main learnings that you as a startup founder will bring with you.
Make data a part of your culture, and let data-driven decisions reduce office politics.
Extensively track marketing metrics to tweak and assure your return on investment. Be creative and try new marketing channels, but be disciplined about their respective performance.
Use research and data to assure yourself that what you're building will result in the desired effect on your core metrics, if not, tweak it until you get there.
Carefully set an objective and focus your energy on the one or two metrics that get you there.
Jason at Spotify summarizes the discussion really well when he says "Rely on data whenever possible. Don’t have enough data? Get more. Make data the most important asset you have because it is the only reliable decision maker that can scale your company".
It seems like we’ve found the secret sauce to growth for early-stage startups. The result of our thesis can be seen in the title, and it is supported by a correlation of 80%, so fairly strong! But let’s take this from the beginning: It all started...
You might be running a SaaS startup or a consumer subscription service, no matter the case, here follows a guide on how to break it down piece by piece to identify the metrics that are important for you. The Action Metrics Tree is optimal to...
You might be running a C2C marketplace for selling off your old CDs or a B2B marketplace to exchange industrial spare parts, no matter the case, analyzing metrics is crucial to achieving growth. Some of the world’s biggest startups...
Ability to retain customers over time is a good indicator of product-market fit and general business health and viability when looked at in cohorts. It’s a vital metric that every startup should track, but can be difficult to interpret off the bat. So...