Introduction to Cohort Analysis for Startups

Cohort analysis: retention

One of the best metrics for identifying whether users love your product is retention. It tells you whether the visitors, which you worked so hard to acquire, are going to come back. Retention evaluation nicely highlights the advantages of a cohort analysis. Unfortunately, while this startup metric can be invaluable, cohort analysis in Google Analytics requires some work to set up.

The graph below shows the daily distribution of visitors who start using your product in each of three separate weeks.

There are numerous ways to calculate the week of the year. The week index reported by Google Analytics is based on a system that’s common in North America: each week starts on a Sunday and the first week of the year is the one containing a Saturday. This is in contrast to the ISO 8601 date and time standard popular in many countries (including most of Europe) that specifies Monday as the start of the week and the first week of the year as the one containing the first Thursday.

Take a look at the blue line in the above graph. When the new week starts, visitors are tagged and counted as members of the Week 43 cohort. By mid-week, the number of unique visitors to your product tends to rise as some of those acquired earlier in the week come back. Depending on your traffic, the cohort count may peak on the last day of the week. On the next day, visitors begin to be assigned to the next cohort group (Week 44), shown in orange above. Without the constant injection of new visitors each day, the visit contribution from the Week 43 cohort falls off dramatically and then decays slowly. Keeping this decline in check is what retention is all about. Retention analysis makes the long-term value of the newly acquired visitors abundantly clear.

There are better ways to visualize how well you’re retaining new visitors. In the cohort example below, each interval shows the users retained as a percentage of the initial visit.

What does a graph like this tell us? First of all, it says that a large percentage of the first-time users of our product do not return the day after they try it. After that, users tend to leave at a much lower, but consistent rate. In other words, if a user returns once or (better) twice then they are reasonably likely to keep doing so. As it turns out, this is a reasonably common observation that’s valid across a number of product types and analysis intervals.

For example, the graph below is taken from a Mixpanel blog post on the importance of tutorials in social games.

Each line represents the weekly retention of a different social game. Obviously the one week retention varies significantly among these titles. Notice, however, that the retention is mostly flat after that first week. Again, initial engagement is critical for long-term retention.

Cohort analysis: engagement

Grouping users into cohorts based on their signup date can also be used to measure changes in engagement. It’s much easier to make informed decisions about new features and refinements when you can correlate cohort behavior changes in your startup analytics with product alterations. For example, you can determine if users that sign up now are more likely to tell a friend about your product than users that signed up a month ago. By combining this with a split test, in which you, say, offer a new sharing feature to only half of your users, you’ll better understand whether improvements can be attributed to the introduction of the new feature.

Let’s look at a cohort data analysis example. In early November you decide to remove a few seemingly redundant instructions from your web service explaining how users can connect and share with their friends. Now you want to ensure that the change was benign. The graph below is a non-cohort trend of your referral rate. This metric consistently hovers around 30%, showing that the changes made to your product during this time have not had a significant effect on referrals. Phew!

However, while it may be true that the overall referral rate hasn’t changed significantly, it’s possible that your large returning user base (of visitors who read your referral instructions before they were removed) are masking the impact of your change. While there are other approaches, the cleanest way to chart the impact of your change is to use a basic cohort analysis. Instead of plotting the percentage of users who refer their friends each week, we’ll instead graph the percentage of users who sign up each week and then, at some point in the future, decide to refer their friends. This allows us to consistently focus on the experience of new users who aren’t tainted by prior experience with the product.

Looking at the cohort for September 1st in the graph above, we would read the data like this: 30% of users who signed up during the week of September 1st eventually decided to refer their friends. There’s an important distinction between this and the previous graph: the users from each weekly cohort who refer friends may do so in the week that they join or any week thereafter. This means that the values for each weekly cohort will continue to change over time.

This is different from cohort retention analysis where the period over which a user may return is usually fixed. Of course, you should feel free to limit the engagement analysis window to whatever is appropriate for your application. It’s possible, for example, that friend referrals from a weekly cohort are only valuable in your product if they occur within 2 weeks of the user signing up.

The cohort graph reveals a couple of things that weren’t obvious in the non-cohort analysis:

  1. Starting the week of September 22nd, the referral rate of each weekly cohort consistently exceeded the average referral rate. Is this statistically significant? What changed? If it’s something you did then you need to do more of it.
  2. Something tragic happened in early November. Perhaps your change wasn’t as benign as you thought or maybe the correlation is coincidental. In any case, there’s a clear need to investigate further.

In both cases, a key performance metric for your startup changed substantially but it was masked by the size of your existing user base. In the same way, it’s possible for a large proportion of new users to mask changes in your existing user base. The cohort analysis uniquely clarifies these changes by tuning your focus to the user groups that provide the most insight into the impact of product changes. 

Next: How to do Cohort Analysis in Google Analytics

Advertisements

48 thoughts on “Introduction to Cohort Analysis for Startups

  1. Pingback: How to do Cohort Analysis in Google Analytics | Jonathon Balogh

  2. Pingback: Quora

  3. Pingback: Citação – Michael Karnjanaprakorn « Startup Biz Model

  4. Pingback: How to do Cohort Analysis in Google Analytics | Jonathon Balogh

  5. Pingback: Founders Dictionary | Cohort Analysis

  6. Pingback: Cohort analysis in a nutshell | A blog about behavioral economics, crowdfunding, crowdsourcing, gamification and more

  7. Pingback: Primer to Building Products in a Lean Startup » taigeair taigeair

  8. Pingback: Web App Analytics Checklist | Dirty AnalyticsDirty Analytics

  9. Great article! Many thanks for sharing. Just noting a small typo in the 3rd paragraph. It should be: “On the next *week, visitors begin to be assigned to the next cohort group (Week 44), shown in orange above.”

  10. Simple and great explanation! Thanks

    How was this visualization generated , the one tagged “The week 3 cohort appears to be more likely to stick with your product. Why?” ?

  11. It’s awesome to pay a quick visit this web site and reading the views of all mates
    regarding this post, while I am also keen of getting experience.

  12. Introduction to Cohort Analysis for Startups | Jonathon Balogh
    Hi there! I just wanted to ask if you ever have any problems with hackers?
    My last blog (wordpress) was hacked and I ended up losing months of
    hard work due to no backup. Do you have any methods to prevent hackers?

  13. Introduction to Cohort Analysis for Startups | Jonathon Balogh
    I think the admin of this site is really working hard for his website, because here every data is
    quality based stuff.

  14. I do not even know how I ended up here, but I thought this post was good.
    I don’t know who you are but definitely you’re going to a famous
    blogger if you aren’t already 😉 Cheers!

  15. Professionally established, designed, and marketed social media accounts such as Twitter and
    Facebook allow skin care companies to build out their skin care communities.
    Before diving into the world of web development, get in terms with jargons of the industry and there is much more to consider than determining
    price. When you go to a website, you instantly get a feel for the business.

  16. What i don’t understood is in fact how you’re
    no longer actually much more neatly-favored than you
    may be right now. You are very intelligent. You recognize thus considerably
    when it comes to this topic, produced me in my opinion consider
    it from numerous various angles. Its like men and women aren’t involved until it is one thing to
    do with Lady gaga! Your individual stuffs nice. All the time take care of it up!

  17. Hello there! This blog post could not bbe writtfen any better!
    Looking aat this arrticle reminds me of my previous roommate!
    He contfinually kept preaching about this. I aam going to send thhis article to him.
    Pretty sure he’s going to have a great read. Many thanks for
    sharing!

  18. The debris container is easy to empty; there are no bags.
    As I rinsed a couple of years of dust down the drain it was hard to not have a little splash up and become airborne.
    Also, due towards the reality which the suction ability of this
    Dyson DC25 all floors vacuum cleaner is fairly impressive, it has the capacity of
    selecting up any form of filth from all varieties
    of floors (that incorporates carpets, rugs, wood, and tiles) with ease.

  19. you are actually a just right webmaster. The site loading speed is incredible.
    It sort of feels that you’re doing any distinctive trick.
    Moreover, The contents are masterpiece. you have performed a great process in this matter!

  20. Excellent post. I was checking continuously this blog and I’m impressed!
    Very useful information specifically the last part :
    ) I care for such information much. I was seeking this certain information for a
    long time. Thank you and best of luck.

  21. Excellent beat ! I would like to apprentice while you amend your site, how could i subscribe for a blog site?
    The account aided me a acceptable deal. I had been tiny bit acquainted
    of this your broadcast offered bright clear idea

  22. Thanks for a marvelous posting! I definitely enjoyed reading it, you are a great author.
    I will make certain to bookmark your blog and will come
    back very soon. I want to encourage you continue your great job,
    have a nice afternoon!

  23. Very nice post. I just stumbled upon your blog and wanted to
    mention that I’ve really loved browsing your blog posts. In any case I’ll be subscribing for your rss feed and I am hoping you
    write once more soon!

  24. Pingback: Player Profiling – A Short Reading List | Game Analytics Resources v. Anders Drachen

  25. Hi blogger, i found this post on 16 spot in google’s search results.
    You should decrease your bounce rate in order to rank in google.
    This is major ranking factor nowadays. There is very handy
    wp plugin which can help you. Just search in google for:

    Lilas’s Bounce Plugin

  26. So, it is necessary that you plan your shoe budget before buying
    one. All footwear at Bourne is hand-crafted especially by
    Opanka construction and the use of adhesives and machinery is minimal while creating the
    shoes, thus making the designs eco-friendly. The Fashion Industry is
    totally focused on how people look and larger people do not figure on mainstream
    fashion’s radar.

  27. Pingback: Metrics, Metrics: Measure to optimize - How to Web

  28. Also, they have ruled the market due to their anatomical constructions of shoes that
    make the shoes anti-stress with special shock absorption system.
    You should always seek advice from someone
    who has purchase the original boots before. Toe Size: The right shoe will not only fit well but will also be flexible in the toe area, with enough
    space for your feet to relax, without being pinched in very tight.

  29. Hola! Gracias poor esta información. Llevaba un rato buscando por laa red sobre este tema
    hasta que he encontrado tu página web. Está muy bien el artículo.
    Prosigue de esta forma!

  30. Metal is the element of this sector, and would
    attract positive Ch’i. If (based on their individual
    energies) most of the party is fire (as an example), we may want
    to downplay the color red. Water fountains are basically put under the water just to make the overall premises stunning or lucrative.

  31. Pingback: Understanding Cohort Analysis - Startups & Marketing Blog

  32. Pingback: User Behavioural Analytics – Yong Liu – Writing for Learning and Sharing

  33. First off I would like to say superb blog! I had a quick question which I’d like
    to ask if you do not mind. I was interested to know how you center yourself and clear your head before writing.
    I’ve had a difficult time clearing my mind in getting my
    ideas out. I truly do enjoy writing but it just
    seems like the first 10 to 15 minutes are wasted just trying to figure out how to
    begin. Any suggestions or hints? Cheers!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s