As Engineering teams start spending more time and effort on incident response, they are usually focused on improving process with their specific team. We think there are additional benefits that can come from a holistic approach to improving incident response across your organization. In this post, we will explore how you can enable Engineering and Customer Success teams to work more effectively when an incident occurs.
Give Customer Success a Voice
One thing that we frequently see within organizations is what’s called “Pitchfork Alerting”.
Pitchfork Alerting [pitch-fork uh-lur-tuhng]
1. When customers raise an incident to a customer success team.
The worst (but inevitable) scenario is when a high priority customer reaches out saying that something is broken. When this happens, they might check your status page but most likely they’re calling their Customer Success Manager alerting them to the incident. More often than not, it’s multiple customers calling multiple managers, and there isn’t an easy way to escalate that incident to the engineering team.
By creating a process that enables the Customer Success team to easily communicate critical information with Engineering, you can reduce MTTA and MTTR without any additional Engineering work.
Shameless plug: FireHydrant enables Customer Success to kick off incidents within Slack and notify the proper Engineering teams.
Include Customer Success in your Engineer’s Runbooks
Runbooks are a magnificent thing. Guiding your engineers through steps needed to remediate an incident helps significantly reduce MTTR and expand your pager rotation. While your engineers are running through their steps, Customer Success is in their own Slack room chattering about the incident and a list of action items. On top of that, customers are at their door asking for details and updates.
By creating role-specific runbooks for teams outside of just engineering, you can empower your Customer Success team to communicate more effectively with their customers. Runbooks enable you to script how Customer Success teams respond to specific customers or how they handle different types of outages which results in a more clear and consistent response. With FireHydrant’s runbooks, it can also enable these users to be added to the proper slack channels, track and communicate remediation, and even email groups of customers.
Pro Tip: You can include sales in this too to help them manage POCs during outages.
Track Customer Impact to Avoid Churn
One of the best things you can do when incidents are happening is track customer impact. We’ve seen it often where each incident gets its own spreadsheet of data that is hard to understand after the fact.
When you use technology that tracks your incident from start to finish and is logically tagged, you get insight on what customers are being impacted and how frequently. This data can then be leveraged during QBRs or during Hot Customer calls to get ahead of churn and improve the customer experience. As a Customer Success Manager, you might use tags like customer name, segment, or impacted high use features.
For the engineers reading this, tags don’t just have to be customers. They can also be microservices or vendors- anything you want to correlate to incidents.
Teamwork helps elevate organizations to achieve higher levels of retention, communication, and efficiency. While there is no single way to help teams work more effectively together, thinking critically about how your engineering team and customer success interact during an incident will help your teams improve on metrics and build a more resilient organization.