Preventing Retry Storms in AI-Assisted Microservices

1

Title: Preventing Retry Storms in AI-Assisted Microservices

Date: July 16, 2026

Speaker: Sibasis Padhi

Registration Link

Modern microservices depend on retries to survive failures. When used carefully, retries improve reliability. When used without limits, they can turn a small problem into a major outage. As organizations adopt AI-assisted operations and autonomous agents, this risk becomes even greater. An AI system may detect a problem and respond exactly as designed, but if every service retries at the same time, the result can be overloaded dependencies, growing queues, rising costs, and cascading failures across the platform.

This talk challenges a common assumption in distributed systems; that more retries always improve reliability. Using real-world examples from cloud-native financial systems, we will explore why retry storms happen, how they spread, and why traditional approaches often fail under pressure. We will discuss practical techniques such as retry budgets, backoff strategies, request deadlines, observability signals, and feedback controls that help keep systems stable during failures.

The session will also examine how AI-assisted operational systems can make better decisions when they are guided by clear boundaries and system-level feedback rather than simply reacting to local failures. As systems become more autonomous, reliability will depend not only on making smarter decisions, but also on knowing when not to act.

Attendees will leave with practical patterns for building resilient microservices that remain stable even when failures, automation, and AI-driven decisions collide.

2

An early example of this kind of failure (though not AI) might have been persistent single path route-flapping resulting from use of packet losses as a cost metric with Dykstra SPF routing where a temporary overload on one link would cause packet drops causing packets to be re-routed, overloading that link resulting in the route changing back… As it turned out, in one case at the ip layer the packet loss measurement also had an issue resulting in a permanent situation that only a restart of a router could fix. Perhaps as relevant might be BART where the emergency stop button signal was originally routed through the computer system, the same computer system that at one point failed to slow down the train at the end of the line. AI systems are not human, and cannot be expected to make judgements on what kinds of failure are more or less acceptable. But the synopsis suggests guidance is being provided on how to fail “less”, and that seems a very good start. Looking forward to it.