Zendesk has been struggling with reliability from it’s beginning - in many ways it has been a victim of its own overnight success. Over the last few years we’ve had to take drastic measures to address major outages, such as implementing company-wide change freezes.

These measures hurt when you have 1000 engineers in 120 product development teams across the globe, and in many ways create more risk when the freeze begins to thaw.

In order to avoid these freeze’s we have recently moved to implement concepts from the Site Reliability Engineering (SRE) discipline, specifically implementing Error Budgets along with SLOs/SLIs. The aim of this is to “scope” the freeze to those systems that have more reliability issues.

We’ve had some wins in introducing this approach, but are still very much at the beginning of this journey. This talk will tell the story of this journey along with providing some practical suggestions around tooling and practices to implement.

Zendesk has been struggling with reliability from it’s beginning - in many ways it has been a victim of its own overnight success. Over the last few years we’ve had to take drastic measures to address major outages, such as implementing company-wide change freezes.

These measures hurt when you have 1000 engineers in 120 product development teams across the globe, and in many ways create more risk when the freeze begins to thaw.

In order to avoid these freeze’s we have recently moved to implement concepts from the Site Reliability Engineering (SRE) discipline, specifically implementing Error Budgets along with SLOs/SLIs. The aim of this is to “scope” the freeze to those systems that have more reliability issues.

We’ve had some wins in introducing this approach, but are still very much at the beginning of this journey. This talk will tell the story of this journey along with providing some practical suggestions around tooling and practices to implement.

Envato runs one of the world’s highest traffic websites, themeforest.net, which is 1 of the 8 sites served by a 2-sided e-commerce platform called Envato Market. Our engineering team juggles serving around 30,000 requests per minute whilst automatically deploying changes once every 30 minutes across a team of 50 developers that contribute to the one codebase. This talk gives the audience a view inside the team’s practices, tooling and culture that allows us to achieve this and maintain high availability.

Zendesk has been struggling with reliability from it’s beginning - in many ways it has been a victim of its own overnight success. Over the last few years we’ve had to take drastic measures to address major outages, such as implementing company-wide change freezes.

These measures hurt when you have 1000 engineers in 120 product development teams across the globe, and in many ways create more risk when the freeze begins to thaw.

In order to avoid these freeze’s we have recently moved to implement concepts from the Site Reliability Engineering (SRE) discipline, specifically implementing Error Budgets along with SLOs/SLIs. The aim of this is to “scope” the freeze to those systems that have more reliability issues.

We’ve had some wins in introducing this approach, but are still very much at the beginning of this journey. This talk will tell the story of this journey along with providing some practical suggestions around tooling and practices to implement.