For smaller companies, you should always aim to be more data-driven. You probably don’t utilize the data you collect (or even collect them at all). The more data you have, the better hypotheses you will generate.
Article in Tech in Asia | Video Highlight
A few months ago, I got a chance to speak at the Tech in Asia Product Development Conference in Jakarta about what it means to be data-driven vs data-informed. After the conference, a few people asked me if I could discuss and speak more about the same topic at other events. So, I thought I might as well share my thoughts.
The term data-driven has started to become a buzzword, stacking on top of a list of other tech jargon. So, let me first get us all on the same page.
Being data-driven means that data is at the center of a team’s and company’s decision-making process. Decision makers operate mostly (and sometimes solely) on data, and some decisions can be made without humans involved.
Here are some things you commonly hear from a data-driven company:
AD. Remove this ad space by subscribing.
The truly data-driven organization will implement this process across teams and functions. This means that every team (product, marketing, customer success, operations, etc.) uses data intensively for all their decisions.
As humans, we have many cognitive biases. In fact, my mentor, Misha Chellam, created flashcards to help people memorize them all.
Data helps get team members and stakeholders on the same page with less of their own judgment.
How many of you have been in a three-hour meeting where people argue back and forth with no sign of compromise?
Sometimes, we discuss and argue about something only for the HiPPO (highest paid person’s opinion) to make the final decision. (Yep, I know how you feel.)
AD. Remove this ad space by subscribing. Support independent journalism.
An MIT study has proven that being data-driven has its benefits. Professor Erik Brynjolfsson and his colleagues studied 179 large, publicly traded companies and concluded that those companies are 5 percent more productive and profitable than their competitors.
However, being data-driven also has its drawbacks.
You have to watch out for outlier distortion. Outliers are data points that are radically different from the average. In gaming, positive outliers are considered “whales.” These are customers who make either big purchases or make them very frequently, generating a significant amount of revenue for the company. At my previous company, we even coined the term “whale hunting.”
In other industries, these might be those customers who use your product to solve a different problem than most of your customers. Their behaviors are so different that they should be excluded from the data you use to make decisions about your average customers. A data set with just a few data points and some outliers can lead your company where it shouldn’t be going.
For startups and big organizations that have just started a data science department, it might be hard to go completely data-driven because of a lack of capabilities and resources. Your current employees might not have enough knowledge to build the infrastructure or you might not even have anyone who has expertise with data.
At my previous company, we had 10 data scientists. At Kulina, we only have 0.5 people with expertise (myself and our head of tech combined).
Most of time, what people say isn’t what they do. So, if we gather data by asking them implicitly, we risk making decisions on the wrong information. For example, during the scandalous time of Uber and the #deleteuber movement, most of my friends said they would never use Uber again. When Lyft (its main competitor in the US) implemented a surcharge, they went back anyway!
AD. Remove this ad space by subscribing. Support independent journalism.
Another great example is when we were trying to figure out which social media platform we wanted to spend more time and money on to engage our users. We sent out a survey asking them, and the result was unexpected.
We saw Facebook on top of the list, which was expected. However, what puzzled us was the fact that Google+ was second on the list (no hard feelings, Google).
A few team members asked if they should spend more time building our Google+ presence. “Maybe we didn’t know our users well,” a team member mentioned. Being data-driven, I would have probably greenlighted this. But being stubborn as usual, I discussed with another product marketing manager on what could’ve gone wrong.
We thought perhaps that our users might not have understood the difference between Google and Google+. In fact, when we looked deeper into the way the question was asked, we found some clues to support this. Instead of using the Google+ icon, we used the word Google+ in the questionnaire. If you didn’t pay close attention, you probably would have thought we were asking if you used Google!
Being data-informed means using data as only a factor in decision making rather than using it as the entire basis. This kind of decision making allows other factors such as customer experience, gut feeling, brand consistency, and the HiPPO to take the lead.
The benefit is that data is now challenged. We don’t purely base our decisions on data but use it to create a hypothesis. This is because no matter how good data is, it has its limitations. It’s just a snapshot of reality that doesn’t paint the full picture of our customer journey and behavior.
By being data-informed, we are forced to use other factors to help make better decisions, including our own judgment.
In my previous gaming company, for example, we always had great revenue on weekends, as that’s when our users played games. But on a particular Monday, one of the new product directors approached me regarding that weekend’s low revenue. He was very worried.
I suspected that our revenue was down because we ran a sale the prior weekend. The same effect occurred for the past five times we ran sales. If we looked at the same cycle before, our average was actually better.
In fact, Facebook’s News Feed was actually implemented without being data-driven. Facebook product designer Adam Mosseri and his team back then decided to create the News Feed without data. The change got a lot of negative reactions, and many users were angry. Some even created a Facebook group titled I automatically hate the new Facebook homepage.
But according to Mosseri, “At the end of the day, we have to have a gut to make bold decisions without data.”
But this doesn’t mean that data-informed decisions don’t have their drawbacks.
Being data-informed requires a lot of time, discussion, and analyses before making a decision. Conclusions also can’t be formed easily. But remember that at the end of the day, “All the data in the world won’t fix a fundamentally bad product,” says former SoundCloud senior product manager Andy Carvell.
In order to be innovative and build the right product for our customers, we cannot just sit in the room looking at rows of data and charts.
Gathering qualitative feedback by talking to your customers, observing their struggles through usability tests, and figuring out the jobs customers are hiring you for are key to building products.
For smaller companies, you should always aim to be more data-driven. You probably don’t utilize the data you collect (or even collect them at all). The more data you have, the better hypotheses you will generate.
For larger companies, you should be able to make bold decisions that not only move your key metrics but also improve customer experience. We should use data to help us make better decisions, but not rely on it 100 percent.
This was first published on Medium
.jpeg)
.jpeg)