Reliability & SRE
Most of what I know about reliability I learned by being on call for it. At TubeMogul I helped take ad delivery from inception to roughly 350 billion daily bid requests, through an IPO and an acquisition; today I lead SRE, Security and Cloud Engineering for Adobe’s Real-Time Customer Data Platform. Both jobs come down to the same thing: systems fail, and the interesting question is what the system does next.
The posts here approach that from two directions. The conceptual ones argue for designing around failure rather than against it — resilience engineering on chaos engineering, FMEA, failure domains and cell architecture, and the algorithms every SRE should learn on the primitives that availability actually rests on.
The rest are case studies. Self-healing techniques used by SRE covers automated failover, forecasting and anomaly detection in production, and is the practitioner version of the IEEE ISSRE 2018 paper that came out of the same work. The two leap second posts are a matched pair worth reading in order: what the 2012 leap second did to the internet, then how we prepared for the 2015 one. A scheduled, announced, well-understood event still took down half the web. That is the whole lesson.
More of this work is written up in my talks and papers.
8 posts
Embrace Disruption: How Resilience Engineering Makes Your Systems Stronger
Discover how resilience engineering, including chaos engineering and FMEA, strengthens systems, turning disruptions into opportunities for growth and adaptability.
Top Key Algorithms that Every Site Reliability Engineer Should Learn
Exploring the importance of algorithms as a critical knowledge of site reliability engineering for ensuring high availability and scalability of software systems.
How To Scale ML Inference to Improve Reliability, Speed, and Cost Efficiency
A practitioner's survey of the tools and strategies for scaling machine learning inference — covering NVIDIA Triton, TorchServe, ONNX, PyTorch compilation, OpenAI Triton, and GPU orchestration on Kubernetes.
Top 5 Machine Learning and Self-Healing Techniques used by SRE
Applying Machine Learning and Self-Healing techniques to the day operations of a production system has become common practices for most SREs. This post cover some real production use cases like automated failover, forecasting, anomalies detection, risk classification and so on.
Getting Ready For The Leap Second
A new leap second will be introduced on June 30th, 2015. This post cover what is the leap second, how the same event crashed most of this internet in 2012, and how you can be ready for it to prevent any outage.
Impact of a Positive Leap Second Introduced In June
Rundown of the impact of the positive leap second introduced in June 2012 and that crashed most of the internet for a full day.
Graphing Java JMX Object values with Ganglia, Python, and JPype
Legacy post on how to graph Java JMX objects with Ganglia and Python.
Cheap SAN and secure backup solution for small-sized platform
Legacy post on how to build a cheap SAN solution with Raid 6 and netwrok block devices (ndb).