Medium’s metric that matters: Total Time Reading
How we measure engagement
One million page views!
50,000 signups!
Five million posts!
165 million active users!
Web companies like metrics — especially when big numbers can be used to woo the tech media into writing about us.
Away from the publicity glare of the Valley tech blogs, every web company should have some not-so-bullshit metrics that guide the business and provide an indication of its health. Ideally, there is one number to rule them all. Josh Elman calls this The Only Metric That Matters.
At Medium, our number is Total Time Reading, or TTR.
The Only Metric That Matters
Let’s first take a step back. Why have a number at all? And if you accept that numbers are a good way by which to measure the success of a business, why have only one?
Away from internet-based companies, most businesses measure their success in dollars. But the media industry has always been a little different. Typically, advertisers pay based on the size of an audience. Various techniques have been used to measure audience size: Radio used diaries, in which listeners would write down what they listened to, and when. Print media added up the total number of copies that were distributed or sold, and then made a guess at how many people saw each copy.
When the web took hold (and e-commerce was just a glint in its eye), only events — like page views and, later, clicks—could be measured. With the widespread use of cookies (and Google Analytics), we progressed to talking about users. For non-revenue-generating start-ups, users were the only currency: registered users, sign-ups, and finally active users.
“Big data” has brought with it the luxury of being able to measure any (and every) interaction that a user has with an application. We can record what a user does, with what device, when, and for how long. The data is cheap to store and relatively easy to process.
We’ve crossed a point at which the availability of data has exceeded what’s required for quality metrics. Most data scientists that I meet tell me that they’re gathering way more data than they can ever hope to use. And yet, in many cases, they still don’t have useful metrics.
In chart form:
Businesses (those with revenue models) are still optimizing for money. Today’s wealth of data helps to better understand what is driving their revenue. Data analysts can join the dots between the earliest user interactions (like marketing campaigns, referral sources, etc.) and end-of-funnel activities (such as spending money or clicking an ad). The data can also provide insight into product diversification or potential new revenue streams.
Companies that don’t have revenue still need to optimize for user behavior that is still valuable. In Medium’s case, that valuable behavior is engaging our users on our platform.
Engagement
Engagement has been the buzzword of growth marketers for a couple of years. When a user engages with your platform, you have their attention. And attention is the precious commodity of the super-connected era.
I think of competing for users’ attention as a zero-sum game. Thanks to hardware innovation, there is barely a moment left in the waking day that hasn’t been claimed by (in no particular order) books, social networks, TV, and games. It’s amazing that we have time for our jobs and families.
There’s no shortage of hand-wringing around what exactly “engagement” means and how it might be measured — if it can be at all. Of course, it depends on the platform, and how you expect your users to spend their time on it.
For content websites (e.g., the New York Times), you want people to read. And then come back, to read more.
A matchmaking service (e.g., OkCupid) attempts to match partners. The number of successful matches should give you a pretty good sense of the health of the business.
What about a site that combines both of these ideas? I sometimes characterize Medium as content matchmaking: we want people to write, and others to read, great posts. It’s two-sided: one can’t exist without the other. What is the core activity that connects the two sides? It’s reading. Readers don’t just view a page, or click an ad. They read.
At Medium, we optimize for the time that people spend reading.
Measuring reading time
TechCrunch’s Gregory Ferenstein wrote:
In fairness to news editors, we do know how much time readers spend on an article: We know that less than 60 percent will read more than half of an article, and a significant slice won’t read anything at all.
I think this is optimistic. It is true that Chartbeat’s analytics will tell you how deeply users engage with content. By their data, on average fewer than 60 percent of users read more than half an article. We see it differently: for us, there are no average users, and there are no average posts.
We measure every user interaction with every post. Most of this is done by periodically recording scroll positions. We pipe this data into our data warehouse, where offline processing aggregates the time spent reading (or our best guess of it): we infer when a reader started reading, when they paused, and when they stopped altogether. The methodology allows us to correct for periods of inactivity (such as having a post open in a different tab, walking the dog, or checking your phone).
The aggregate Total Time Reading (TTR) is a metric that helps us understand how the Medium platform is doing as a whole. We can slice that number in lots of ways (logged-in vs. logged-out, new posts vs. old, etc.).
We’re thinking about other ways in which this data can be used to learn about Medium users — and their interactions with specific posts. For example:
- How can we motivate users to increase the total time spent reading the posts that they’ve written?
- We measure the length of posts in Expected Reading Time. So, which is better: a user spending three minutes reading half of a six-minute post, or a user spending two minutes reading a two-minute post?
- If a user spends four minutes reading a six-minute post, did she skim it? Is she just a super-fast reader? Or is our time estimate wrong?
- How long does it take the eye to register an image?
- What’s the optimal length of a post if we want to maximize TTR?
And so many more.
