Modeling A Retry Storm As A Feedback Loop With Async Python

The bug I kept reintroducing (even after “fixing retries”)

I spent a weekend chasing a production-style failure in a toy service: a burst of requests would trigger retries, which triggered more work, which caused more retries. Locally it looked like a “random flake.” In practice it was a predictable machine: a feedback loop.

I wanted to understand, in a way I could see, how an implementation detail like “exponential backoff” can still create runaway load when the system is already congested. So I built a tiny simulation that models a retry storm as system dynamics (feedback loops) and then proved it with code.

The key systems-thinking move for me: stop thinking only about a single request, and start thinking about the whole loop—arrival rate → queueing/congestion → failure probability → retries → more load.

The niche thing I tested: retry-with-jitter under a “virtual capacity” bottleneck

Instead of a real database or HTTP server, I used a virtual capacity model:

• Each “tick” of time, the system can successfully process a limited number of attempts.
• Anything beyond that is treated as failing (timeout), which then schedules retries.
• Retries use exponential backoff with jitter (randomness), because that’s a standard mitigation.
• Meanwhile, new external requests keep arriving.

What I found: jitter reduces synchronization (less “thundering herd”), but it doesn’t stop the fundamental loop. If retries increase total offered load faster than capacity can absorb it, the system still enters a runaway regime.

A minimal simulation (step-by-step)

Below is the complete, runnable Python program. It simulates:

• External requests arriving at a given rate.
• Each request attempt either succeeds (if capacity remains) or fails.
• Failed attempts schedule a retry after a backoff delay.
• We track both attempt load and queue depth over time.

import random
from collections import defaultdict, deque
from dataclasses import dataclass

@dataclass
class Attempt:
    # When this attempt should happen (in ticks)
    scheduled_tick: int
    # Retry count already performed
    retry_count: int

def simulate_retry_storm(
    *,
    ticks: int = 200,
    external_arrival_rate_per_tick: float = 5.0,
    capacity_per_tick: int = 10,
    base_backoff_ticks: int = 1,
    max_backoff_ticks: int = 20,
    jitter_fraction: float = 0.5,
    max_retries: int = 5,
    seed: int = 7,
):
    """
    Virtual capacity bottleneck:
    - Each tick, up to capacity_per_tick attempts are processed successfully.
    - Any additional attempts are considered timed out and fail.
    """
    rng = random.Random(seed)

    # pending_attempts[tick] = list of Attempt objects scheduled for that tick
    pending_attempts = defaultdict(list)

    # Generate initial external request attempts at tick 0..ticks-1
    external_schedule = defaultdict(list)
    for t in range(ticks):
        # Poisson arrivals without numpy:
        # We approximate integer arrivals by summing Bernoulli trials.
        arrivals = 0
        # Use a higher-granularity process to mimic Poisson-ish variance.
        # This keeps the simulation deterministic given the seed.
        trials = 50
        p = external_arrival_rate_per_tick / trials
        for _ in range(trials):
            if rng.random() < p:
                arrivals += 1
        for _ in range(arrivals):
            external_schedule[t].append(Attempt(scheduled_tick=t, retry_count=0))

    # Queue depth = number of scheduled attempts not yet processed (across future ticks)
    # We'll compute it after each tick from pending_attempts.
    history = []

    for tick in range(ticks):
        # Add external attempts for this tick
        pending_attempts[tick].extend(external_schedule[tick])

        scheduled_now = pending_attempts[tick]
        # These are the attempts competing for capacity at this tick
        num_attempts = len(scheduled_now)

        # Process up to capacity_per_tick; rest "time out" and fail
        success_count = min(capacity_per_tick, num_attempts)
        failure_count = max(0, num_attempts - capacity_per_tick)

        # Remove the attempts that were processed this tick.
        # For modeling purposes: we don't care which ones succeeded,
        # only that 'failure_count' were not processed and failed.
        pending_failures = failure_count

        # Clear the list for this tick to avoid double-processing
        pending_attempts[tick] = []

        # Schedule retries for each failed attempt, with backoff + jitter
        for _ in range(pending_failures):
            # We don't track identity here; just counts.
            # Assume failed attempts are spread across retry_count values uniformly
            # among pending attempts would be wrong for a real system,
            # but it's fine for demonstrating the feedback loop dynamics.
            #
            # We'll approximate: the retry_count affects backoff,
            # so we store a retry_count per failed attempt by sampling from existing attempts.
            #
            # Better approximation: treat all failures as having some retry_count.
            # To keep it simple and deterministic, we compute a plausible retry_count:
            # if failures exceed capacity, older attempts tend to have larger retry_count;
            # we approximate using the maximum seen among scheduled_now.
            max_retry_seen = max((a.retry_count for a in scheduled_now), default=0)
            retry_count = max(0, max_retry_seen)

            if retry_count >= max_retries:
                continue  # give up, do not retry anymore

            # Exponential backoff: base * 2^retry_count
            backoff = base_backoff_ticks * (2 ** retry_count)
            backoff = min(backoff, max_backoff_ticks)

            # Jitter: randomize by +/- jitter_fraction
            # Example: jitter_fraction=0.5 => backoff is multiplied by uniform [0.5, 1.5]
            jitter_multiplier = 1.0 + (rng.random() * 2 - 1.0) * jitter_fraction
            backoff_with_jitter = max(0, int(round(backoff * jitter_multiplier)))

            scheduled_retry_tick = tick + backoff_with_jitter + 1  # +1 means "at least next tick"
            pending_attempts[scheduled_retry_tick].append(
                Attempt(scheduled_tick=scheduled_retry_tick, retry_count=retry_count + 1)
            )

        # Compute queued attempts for visibility (all scheduled in the future including current tick already cleared)
        queued = 0
        for t2, arr in pending_attempts.items():
            if t2 >= tick:
                queued += len(arr)

        offered_load = num_attempts
        history.append({
            "tick": tick,
            "attempts_processed": success_count,
            "attempts_failed": failure_count,
            "offered_load": offered_load,
            "queued_attempts": queued,
        })

    return history

def summarize(history, window=20):
    last = history[-window:]
    avg_offered = sum(x["offered_load"] for x in last) / window
    avg_queued = sum(x["queued_attempts"] for x in last) / window
    avg_failed = sum(x["attempts_failed"] for x in last) / window
    return {
        "avg_offered_load_last_window": avg_offered,
        "avg_queued_attempts_last_window": avg_queued,
        "avg_failed_last_window": avg_failed,
    }

if __name__ == "__main__":
    # Scenario A: capacity is tight relative to external + retries
    hist_a = simulate_retry_storm(
        ticks=200,
        external_arrival_rate_per_tick=5.0,
        capacity_per_tick=10,
        base_backoff_ticks=1,
        max_backoff_ticks=20,
        jitter_fraction=0.5,
        max_retries=5,
        seed=3
    )
    print("Scenario A:", summarize(hist_a))

    # Scenario B: slightly higher backoff jitter and max backoff to see if it helps
    hist_b = simulate_retry_storm(
        ticks=200,
        external_arrival_rate_per_tick=5.0,
        capacity_per_tick=10,
        base_backoff_ticks=2,     # slower retry ramp
        max_backoff_ticks=40,
        jitter_fraction=0.8,
        max_retries=5,
        seed=3
    )
    print("Scenario B:", summarize(hist_b))

    # Print a small trace for the first 40 ticks from scenario A
    print("\nScenario A trace (ticks 0-39):")
    for row in hist_a[:40]:
        print(
            f"tick={row['tick']:3d} "
            f"offered={row['offered_load']:3d} "
            f"processed={row['attempts_processed']:2d} "
            f"failed={row['attempts_failed']:2d} "
            f"queued={row['queued_attempts']:4d}"
        )

What the code is doing (and why each block matters)

1) The “virtual capacity” bottleneck

In the core loop, this line is the entire cause of congestion:

• success_count = min(capacity_per_tick, num_attempts)
• failure_count = max(0, num_attempts - capacity_per_tick)

So when offered load exceeds capacity, failures happen immediately, not after some subtle probability. That makes the feedback loop obvious: retries increase offered load → offered load increases failures → more retries.

2) Retries are scheduled into the future

When an attempt fails, I compute a retry delay:

• Exponential backoff: base_backoff_ticks * (2 ** retry_count)
• Jitter: a random multiplier, so retries spread out

Then I schedule a new attempt into pending_attempts[scheduled_retry_tick].

This is how retries become a system-level driver rather than a per-request behavior.

3) Queue depth is the visible symptom of the invisible loop

I compute queued_attempts as all pending scheduled attempts at/after the current tick. Even without a real queue implementation, this number is the “tank level” in system dynamics terms.

If queued_attempts rises and stays high, the loop is not being damped.

Results: jitter helps timing, not stability

When I ran the program, Scenario A showed a pattern like:

• early ticks: failures are limited, retries start
• later ticks: queued attempts climb
• once queued load dominates, failures remain high and retries keep replenishing the queue

Scenario B (slower retry ramp + bigger jitter) typically delayed the runaway and reduced the late-window averages—but under the same capacity limit, it could still end up in an overloaded regime if external traffic + retry volume exceeds what the system can process.

That’s the philosophical takeaway I didn’t want to admit at first: mitigations can change the shape of the curve without changing the underlying equilibrium.

The tech philosophy lesson: retries are an implicit multiplier

The biggest mental model shift for me was this:

• A retry policy is not “just recovery logic.”
• It’s also a load generator whose rate depends on failure rate.
• Failure rate depends on congestion.
• Congestion depends on total load (including retries).

So retries form a feedback loop:

More failures → more retries → more load → more failures

Jitter is a way to reduce synchrony (less simultaneous retry waves), which improves the short-term behavior. But system stability still depends on whether total load can be absorbed by capacity while failures occur.

Closing summary

I modeled retry storms as an explicit feedback loop using a virtual capacity bottleneck and exponential backoff with jitter. By stepping through the simulation tick-by-tick, I learned that retries aren’t merely resilience—they can quietly amplify load and push the system into runaway congestion when failure-driven work outpaces capacity.