Building a Resilient Infrastructure, One Failure at a Time: Olamide Olaoye’s Approach to Site Reliability Engineering

By Salami Adeyinka

Building resilient infrastructure is an ongoing challenge that requires both technical expertise and a mindset that embraces failure as a learning opportunity. Olamide Olaoye, a Senior Site Reliability Engineer (SRE), has spent the better part of a decade mastering this craft, honing a unique set of skills that blend backend development, cloud infrastructure, and incident management. His journey from a backend engineer to an SRE has equipped him with valuable insights on what makes systems not only reliable but also adaptable in the face of inevitable failures.

Olamide’s career began in 2016 as a backend engineer, working primarily on banking applications and API/mobile app integrations. His first foray into the world of DevOps came when he was tasked with deploying applications to the Azure cloud. The complexity of migrating sensitive banking data and ensuring that these systems ran seamlessly in a cloud environment sparked his interest in infrastructure management. By 2020, Olamide had transitioned fully into an SRE role, leveraging his software development background to approach problems from a unique perspective. Unlike traditional system administrators, who often focus on managing infrastructure, Olamide’s deep understanding of the code behind applications allows him to troubleshoot and resolve issues during production incidents with minimal support from backend developers. This ability has made him an invaluable asset to his team, especially during critical situations.

At its core, the role of an SRE is about ensuring production integrity, high availability, and uptime for customer-facing applications. Olamide’s work is built around these principles, but his approach to resilience goes beyond simply scaling infrastructure to handle traffic spikes or ensuring that systems stay up during outages. He believes that the true measure of a resilient system is its ability to recover from failure. By intentionally designing systems that can withstand failure and recover autonomously, Olamide ensures that downtime is kept to a minimum, even when things go wrong.

One of the most important practices that Olamide embraces is chaos engineering, a methodology that involves intentionally introducing faults into a system to test its resilience. This may sound counterintuitive, but it is a highly effective way to identify vulnerabilities and assess how well a system can recover. Chaos engineering allows teams to simulate real-world failure scenarios, providing them with the opportunity to patch weaknesses before they become critical issues. For Olamide, these controlled failures are key to building systems that can self-heal when disaster strikes.

As systems become more complex, especially with the rise of microservices architectures, observability has become a crucial component of Olamide’s work. Using tools like OpenTelemetry and distributed tracing, he ensures that teams have full visibility into their systems. This allows them to track issues in real-time, diagnose problems quickly, and reduce downtime during incidents. Without observability, even the most resilient systems can struggle when something goes wrong. The ability to see inside a system, diagnose problems, and respond proactively is what makes the difference between a short-lived failure and a prolonged outage.

In recent years, the integration of AI and machine learning into monitoring and incident management has also transformed the SRE landscape. Olamide is particularly excited about the potential of AI-driven tools to predict issues before they occur, optimize resource allocation, and even self-heal in some cases. These advancements have the potential to revolutionize how teams approach site reliability by shifting the focus from reactive incident response to proactive, data-driven problem-solving. Additionally, DevSecOps—the integration of security practices across the entire software development lifecycle, including but not limited to the CI/CD pipelin—has gained significant traction, and Olamide believes it is essential for securing systems at scale. Automation in this area can help prevent security breaches by ensuring that vulnerabilities are detected and addressed before they reach production.

Olamide’s work also extends to the open-source community, where he actively contributes to projects like Flightdeck, a suite of AWS and Kubernetes modules that helps developers adopt cloud technologies more easily. Flightdeck automates the setup of scalable infrastructures on AWS, allowing companies to quickly deploy secure, compliant applications in the cloud. One of Olamide’s other contributions is an open-source AWS automated backup module that scans AWS organizations for backup needs and automatically backs up critical data to protect against disasters like ransomware attacks.

Throughout his career, Olamide has faced many challenges, including a common issue within the SRE field: the lack of software development knowledge among many engineers. He believes that proficiency in at least three programming languages, such as Python, Go, or JavaScript, is critical for success in SRE. This programming knowledge allows engineers to automate redundant tasks, build tools for improved monitoring, and tackle complex issues that require coding expertise. Additionally, insufficient observability and tracing remain significant challenges, and Olamide recommends investing in comprehensive observability solutions to enhance error detection and speed up incident resolution.

Looking to the future, Olamide is excited about the potential growth areas within the SRE industry. GitOps, which uses Git as the source of truth for managing infrastructure, is gaining momentum beyond Kubernetes and could soon be applied to other areas like networking and machine learning workflows. Furthermore, DevSecOps tools and practices are still in their early stages, leaving ample room for innovation in automating compliance and embedding security into the development process. Olamide is also particularly intrigued by the growing interest in building unified observability platforms that integrate logs, metrics, traces, and user experience data to provide a cohesive view of a system’s health.

For Olamide, building resilient infrastructure is not just about applying best practices or using the latest tools; it’s about cultivating a mindset that values learning from failure. As systems become more complex, embracing failure as a part of the learning process will be crucial for the continued evolution of site reliability. In a world where disruptions are inevitable, building systems that can withstand and recover from failures will be the key to ensuring uptime and reliability. Through a combination of proactive measures, automation, and innovative tools, Olamide is helping shape the future of site reliability engineering.