A Behind-the-Scenes Look at How We Release the Spotify App (Part 1)
Developing and releasing mobile apps at scale is a big challenge. With each weekly release of our mobile app for iOS and Android, hundreds of changes go out to more than 675 million users all over the world and on all kinds of mobile devices. A lot can go wrong, so discovering and mitigating potential and confirmed issues is critical to ensuring a smooth listening experience. Every feature could impact the app’s stability and user experience, so making sure we roll them out in a coordinated and prioritized way is also something that has to be taken care of.
At Spotify, our Release team has a dual mission: (1) to oversee the release of the main Spotify app and (2) to build the necessary tools to support this process. Day-to-day coordination is handled by the full-time Release Manager with the support of the rest of the team.
We have a logo, which means we mean serious business
When it comes to releasing the app, the core responsibilities of the Release team are twofold:
Making sure that the time from when the developer merges their code into the main branch to when it’s available to users is as short as possible
To ensure that quality meets our standards
At times, friction exists between these two goals, and much of the craft of release management is about mitigating this, both with tooling and with informed coordination and decision-making. Instances when this balance needs to be struck might include the following:
Prioritizing accordingly — not all bugs are created equal. A crash during signup or playback demands immediate attention, while a post-logout crash might be less urgent.
Identifying a fallback — if a bug affects a specific A/B test group, we can temporarily route all users to the working experience via backend adjustments, addressing the client-side fix in the next release. This keeps the release on track without sacrificing user experience.
Acting quickly when necessary — even a minor bug affecting a small but significant user group (e.g., crashes in a specific region) might warrant a swift resolution.
The release cycle
To illustrate our release cycle, let’s follow the journey of version 8.9.2 from inception to rollout.
Friday, September 20: A new version is born
Each release cycle kicks off on a Friday morning, when the release of the previous version has been cut. Once this has been done, it’s time to start the work on the upcoming version.
At Spotify, we practice trunk-based development, meaning developers merge their code into the main branch as soon as it’s tested and reviewed.
However, we make an exception for major changes: Large-scale or infrastructure updates are merged earlier in the cycle (typically on the first day, Friday of Week 1). This gives us ample time for thorough testing, leveraging both internal teams and external alpha users to identify and fix issues early on.
With Spotify 8.9.2, we planned to roll out the Audiobooks feature in some markets — it had been available behind a feature flag in the backend for a number of releases, for internal testing to discover any bugs that might impede the planned rollout. This was an important new feature for the company, and we wanted to make sure we got it right, particularly since marketing activities and events were already scheduled.
The Release Manager made sure, well in advance, that it was the only big feature rolling out with this release — another new feature that initially had been scheduled to roll out in the same week was rescheduled for the following week.
Other teams could still merge code at any moment, but we strongly encouraged them to use feature flags. If that wasn’t possible, we asked them to avoid merging any high-risk changes on that particular week.
Friday, September 20 – Thursday, September 26
Apart from the additional actions during the first day, each day of the first week of the release cycle basically looks the same.
Early each morning, nightly builds of the main branch are sent out internally and to our alpha users.
Teams develop and merge new code. The developers and their teams make sure that the code is tested and reviewed beforehand.
Bug reports are filed by internal and external alpha users. When the owner of the affected feature is unknown to the reporter, the Release Manager makes sure that the bug report gets assigned to the correct team.
Crash rates and other metrics are tracked for each build both automatically and manually. Automatic bug tickets are created when a crash or other issue exceeds our predefined severity threshold; manual tickets are created when something is deemed worthy of investigation by the Release Manager or any other employee.