How can you go from 0-10 with product analytics? In this article, we are going to be exploring this question whilst offering practical tips on how to build a product measurement strategy.
Fatih Koca is Director of Analytics Strategy at Mixpanel. He advises Mixpanel customers on developing their analytics strategy and improving data-informed decision making. He is also guest lecturing on analytics at various Bay Area universities.
Before Mixpanel, Faith Koca has had more than 10 years in a data-related career, spending the last six years being at eBay in different roles. The final role at Ebay centered around heading up their global marketing analytics team at eBay classifieds group, which is a portfolio of companies across the world.
As a person that has been working with startups, one of the biggest problems that I have seen startup founders facing is confusion over how to set up their analytics strategy from 0 to 1. They might be aware of different metrics and different things that they want to measure, but they really don’t know how to build everything in a more holistic manner.
In other words, they don’t know how to build with the bigger picture of their analytics in mind. This is something that Fatih Koca has extensive experience doing. I was very keen to hear Fatih’s thoughts on how a startup founder could set up their analytics from 0 to 10.
Fatih Koca shared his thoughts with us, “I think that’s a problem for startups and for even bigger corporations. Right now, we have more and more data points coming from users about their behaviors, about their demographics, transactions. So, the data is abundant. In my role, what I see is sometimes the company is collecting whatever data they try to make use of that”.
Fatih continued, “After having lots of conversations with different customers, I recently developed a framework, Mixpanel Measurement Framework to guide our customers. When we look at the data ecosystem, we see that data is flowing from everywhere. Data galore! And then it raises the question, as you mentioned, about how to go from 0 to 1 and 10 and 100. As many things, sophisticated and complex things, frameworks become a great help.
Fatih Koca is a big believer in the power of frameworks: “Frameworks are designed to simplify a complex topic and make it more efficient. We think that a product measurement strategy should start with a focus metric. That focused magic is the magic that matters most. I do not believe the concept of there’s only one metric that matters for the business. It is more like the pigs in Animal Farm. In an Orwellian way, all metrics are equal, but the focus metric is more equal”.
He continued, “This metric should be tied to the overall strategy and this metric should capture the value of the product and the value that the customer is getting from the product. And then we go one level down. Those metrics compliment the focus metric and give a holistic understanding of the user journey all the way from acquisition all the way to retention. First, we have a metric called ‘Reach’, which is basically our user base”.
“How many people are interested in using our product? How many people we targeted through our marketing campaigns and they came and we were able to show them something and they showed interest, but maybe they did not come back and do the repeat action? So, what is the ceiling of the weekly active users that we can activate?”, Fatih said.
“Then we have an ‘Activation’ metric. ‘Activation’ captures the Aha moment, the Eureka moment for the new user or reactivated user. This is a metric to capture how efficient we are to activate, to convert a passer-by into a loyal user. And then we have ‘Engagement’. We want more users and we ideally want them to engage more and then come back and do the same engagement”.
As Fatih told us, “There is no one framework that fits all business use cases. Every business is unique. That’s why we have business specific buckets to capture KPIs that measure the specifics of that business. After figuring out these metrics based on our high level strategy and to have a mutually exclusive and collectively exhaustive way of looking at the user journey, we go one level deeper and start thinking about level two metrics”.
“These become more tactical, more specific. They can be broken down by platform, region, segment, or feature. And then we can go one level down and decide on level three metrics. The important part is first picking the KPIs, picking the right KPIs. You also need to be able to identify owners and champions who can proactively think about ideas and features to improve those key performance indicators for the long term.”
Fatih believes that ownership and accountability is the second important part, and then goals. “You need to have goals for these KPIs so that the owner, the champion, and the people who can influence those KPIs can work to improve those KPIs, to improve the P, which is the performance. The fourth most important thing is the democratization of data. Everyone who is involved in the product and the business should be aware of the relevant metrics that matter for the business”, he explained.
To explain the importance of picking the right KPIs, Fatih likes to use the example of a video streaming platform, such as Netflix. “For a company like Netflix, it is really important to have more subscribers. And it’s really important to make sure that those subscribers are getting value from the product. If they don’t, they might churn in the long term. And if you don’t understand their active usage and the short term, it’s going to be too late by the time when they churn”.
Fatih continued to explain the concept: “Taking a short term look on the weekly active subscribers is going to be helping us to provide value to our most valuable users who bring us revenue. Weekly active subscribers would be a good focus metric for a business like Netflix, and then ‘Reach’, our user base. And in this case, the number of total subscribers would be our total reach”.
“Some Netflix users might be active weekly, some will not, but it will be good to have a closer view on our user base, which can be segmented into new and existing in our top level. The ‘Activation’ metric, as I mentioned, is a metric that captures the efficiency of our product to convert a new user into an active and loyal user”, he told us.
“We want more active subscribers, and we want them to engage more when they get more value, they will churn less. So, minutes viewed, minutes watched, over weekly active subscribers would be a good engagement metric. In level two, how many videos they watch, would be a good level two metric, which can be owned by a product team, the team that is building the recommendation engine”.
When you are picking focus metric, Fatih emphasized that it is really important to define the active part. “‘Active’ should be a hundred percent tied to the value that the product is providing. In Netflix case or in video streaming subscription case active, it is people watching a video. If it was a bank, this metric should be capturing people who are depositing money or getting some credits”, he said.
The second phase is about what to measure. We now have the framework, but how can we flesh this out? Fatih told us that it should start with a high level product strategy. “We should tie our focus metric to our strategy. And then we need to go one level down and, and identify key performance indicators, which measures the performance of executing that strategy. And in the short term, the metrics will come. A KPI might be minutes watched per week. The active subscribers metric could be click-through-rate”.
Fatih continued, “To enable those metrics and KPIs, what data do we need and what tracking do we need? Strategy drives implementation and implementation, a good implementation, underpins the long term strategy. As I mentioned earlier, there are four important things to enable successful execution of measurement strategy”.
“To bring this from theory to practice. The first one is defining the right key performance indicator. The second one is identifying champions and owners of those KPIs. The third one is to set the right goals and the fourth one is making sure his data is democratized. Those metrics are democratized through dashboards, verbal communication, weekly insight meetings, or written communication through emails”.
I’ve seen lots of businesses implementing fancy product analytics tools, such as Mixpanel and having them turn into legacy systems after a couple of months. Many businesses end up getting buried under all of the data and numbers. I wondered whether Fatih had any specific advice to avoid creating another legacy system that nobody’s using.
In response to my question, Fatih believes that relying on this methodology really solves that problem. “Applying this methodology of first thinking about the strategy, what we are trying to accomplish with that, and then identifying KPIs and laying out the implementation plan and thinking about what we need to track. Then surfacing those two Mixpanel dashboards and closely monitoring them in a real simple way”.
“We shouldn’t have twenty, thirty KPIs in one dashboard. We should have six. That’s what the dashboard is. You have high level KPIs. If you are looking at a holistic user journey, where you can put your focus metric. And if you want to go tactical, if you want to build the retention dashboard, then you might go to level two, level three metrics of retention and convert them into a retention dashboard”.
We hope that you enjoyed learning about how to build an effective product measurement strategy to ensure that your business makes data-informed decision making. I very much enjoyed talking with Fatih Koca about this fascinating topic and it was particularly interesting to learn more about Mixpanel’s approach to product analytics.