Sometime in the Spring of 2020 around midnight, Opsgenie alert notified the team of an escalation in one of our critical services. Our team built and owned APIs and services that helped deliver billions of text messages globally 24-7 reliably. A few minutes later, the whole engineering team of 9 were up in fire-fighting mode, Grafana watching, the team Slack was buzzing, suggestions and counter suggestions were thrown and Zoom was fired up. That night, the team had access to everything they required to independently resolve the escalation including but crucially, the motivation. There was a genuine eagerness to understand what went wrong, document resolution steps and plan to permanently fix the cause of the escalation. This was genuinely a team effort.
After each escalation, a team member elects (ownership) to write down the root cause analysis (RCA) document to ensure we understood the causes and where necessary create Tech debt tickets to permanently resolve the cause of the escalation. As a team, we agreed to use up to 30% of our team capacity in tackling tech debts (and other dev initiatives) and 70% to continue building out our product roadmap. As the Technical Delivery Manager, I communicated the tech-debt/product ratio decision to product and leadership teams on the importance of tackling our tech debts in business value terms. For instance, fixing these tech debts will enable us to realise the goal of growing our API ingestion rate by 25% and subsequently deliver more revenue for the business. I ensured the team’s work was visible and its impact and value were adequately communicated and understood across the business. Here, I outline the steps the team applied for efficient management of escalation.
From the events above, the team was empowered and the business genuinely trusted the squad to be self-organising. Empowered teams are trusted teams, trusted teams are motivated teams, and motivated teams are innovative teams. Innovative businesses win customer trust and loyalty and then subsequently generate the revenue targets of the business. As a team, we were able to reliably help the business grow the monthly traffic on our messaging platform from 3B to over 7.2B messages monthly. This subsequently ensured the business’ revenue grew by over 20% YoY with no significant investment in dev resources and infrastructure. Team empowerment is a low-hanging fruit that a company leadership could use to help improve revenue without significant capital investment.
The best architectures, requirements, and designs emerge from self-organising teams.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
Agile Manifesto
Tech innovation is probably impossible without a series of failures and businesses that are looking to launch innovative products must have a failure-tolerant and resilient culture at its core. Most innovations from businesses are the result of iterations and knowledge from previous failures. The outcome of failure should be on the learnings, improvements and risk-containment strategy. Complex technology products are fraught with risks, unknowns and several edge cases that can never be fully understood until something fails. One of the key reasons SpaceX is successfully disrupting space transportation is its high tolerance for failure and every failure brings them closer to achieving its ambitious goals.
The adoption of DevSecOps and other engineering best practices is enabling companies to carefully speed up their time to production while having robust quality gates and security compliance at the heart of the process. Teams and companies with robust logging, monitoring and testing regimes often deploy faster knowing that risks could be spotted and treated quickly. Empowered teams are collaborative, supportive and agile. We were able to rewrite and deprecate a 15-year monolith with microservices and a modern web stack through collaboration among the team members. Junior software engineers made suggestions and the senior developers guided and carefully managed the implementation and decoupling process.
Successful delivery of complex products often starts with leadership’s commitment to achieving positive outcomes and empowering the teams to achieve those ambitious targets. As a leader, have empathy during failure and not apathy while reinforcing your belief and trust in your team. Prof Edward D. Hess summarises the importance of having the right culture; Innovative organisations build the right culture and an enabling internal system that drives innovation behaviours. Along with mindsets and systems come the right experimental processes. But underlying all of this is one key concept: you must be willing to accept failures as a necessary part of the innovation process
An inspiring read. I hope tech companies see and apply these strategies you have documented. Great to see actionable steps especially how companies are using DevOps to speed up their time to production!
Cheers! Please write more!