Lesson 5: Case study: Shuffle button in a shelf on Spotify Home

Write a hypothesis for an experiment that adds a shuffle button to a shelf on Spotify Home

This case study is based on an actual experiment at Spotify. The details have been modified for the purpose of this exercise. The experiment adds a shuffle button to the 'Try something else' shelf on the home screen of Spotify.

The text below gives some information on earlier research, theory, and motivation that led to an experiment which tested the impact of adding a shuffle button on the 'Try something else' shelf on the home screen.

Use the information below to formulate a testable hypothesis for this experiment.

Prior knowledge

  • In earlier user research, we found that Just play something actions decreased the perceived friction to start listening.
  • A large part of the consumption from Spotify is via surfaces that automatically select what audio to play.
  • Anecdotal evidence that the stress of having to choose can explain why users continue to consume the same content.

Theory about user need

Providing the user with low-effort paths in terms of making a decision can decrease the load on the user to start listening, and thereby increase engagement and audio consumption.

Motivation

  • If users get stuck in browsing for content, and are unable to make a decision, they might leave the app.
  • Offering a low-effort path to start listening will decrease perceived stress and can increase engagement and audio consumption from the home screen.
  • Similar features have been successful in other places in the app before.

Exercise: Write a testable hypothesis for this experiment

  1. Which success metrics would you set for this experiment? Why would you choose these metrics?

  2. What unintended side effects could you see when you add a shuffle button to this shelf? What guardrail metrics would you set to test for this?

  3. Based on the information above, write a testable hypothesis for an experiment on a shuffle button on the Try something else shelf. You can use the template below as a starting point.

You can find answers to these questions on the next page.

Hypothesis template

Based on [prior knowledge], we believe that [theory about user need].

We think that [doing this/building this feature/creating this experience] for [these people/personas] will achieve [these outcomes].

We will know this is true when we see [metric results]. This will be good for [customers/artists/ our business] because [motivation].

Bonus questions

  1. How much of an increase in these metrics would you want to see to call the experiment a success?
  2. What is the largest decrease in the guardrail metrics that you would still consider acceptable?

You may not be able to give an exact number for questions 4) and 5), but think about how you would come up with a number to answer these questions.

Reader exercise

What is the primary reason for selecting 'Minutes played on week 1 after exposure' as a success metric instead of 'Consumption from the shelf with the shuffle button'?