# Run Game Days Against Dependencies You Do Not Control

> Exercise bounded failures in DNS, certificates, cloud APIs, payments, email, and identity providers with explicit hypotheses and customer-visible fallbacks.

Canonical URL: https://www.devobs.io/articles/dependency-failure-game-days/
By: Maya Chen
Published: 2024-06-28T17:26:55.240Z
Updated: 2026-09-06T10:18:15.722Z
Section: Architecture

## Define dependency-failure contracts before injecting faults

Start from a contract: timeout, retry behavior, rate limit, consistency, and customer-visible fallback. State a steady-state metric and a falsifiable hypothesis, then inject only enough latency or errors to test the response. The [Principles of Chaos Engineering](https://principlesofchaos.org/) provide a useful operating model for this decision: define steady state, introduce real-world variables, and try to disprove the hypothesis while minimizing blast radius.

## Test payment and DNS failure paths

For a payment provider, inject timeouts after request submission and observe whether idempotency prevents double charge while the UI reports an honest pending state. For DNS, use a test hostname or narrow cohort instead of damaging shared resolution. The [Google SRE book chapter on testing for reliability](https://sre.google/sre-book/testing-reliability/) supports the broader reason to do this work: testing reduces uncertainty about future reliability, and production tests evaluate whether a deployed system is working correctly.

## Design the experiment around a user outcome

State a hypothesis before injecting failure: “If the tax provider times out, checkout remains available for tax-exempt orders and other orders fail within three seconds with a retryable status.” Define blast radius, duration, abort signals, excluded tenants, and the person who can stop the exercise. Use test credentials and a narrow cohort.

Faults should resemble real dependency behavior: slow responses, connection refusal, partial success, [our guide to distributed rate-limit semantics](https://www.devobs.io/articles/distributed-rate-limit-semantics/), invalid payloads, and delayed recovery. A clean HTTP 500 is often easier than the failures production actually creates.

## Watch decisions, not only graphs

During the exercise, record when the first symptom appeared, which alert fired, what the responder believed, and why they changed a timeout, circuit, or feature. Confirm retries do not amplify load and that queued work remains bounded. Verify customer messages distinguish delayed, rejected, and completed requests.

After recovery, reconcile with the provider’s authoritative records. Turn surprises into one owned change with a verification method, not a broad resilience backlog. Start with the dependency whose outage caused the most recent customer impact and run a ten-minute staging exercise that tests its timeout and recovery path. Repeat the recovery phase after removing the fault; half-open circuits, accumulated retries, and stale DNS can prolong impact after the provider returns.

Review date: 2026-09-06.

## Source references

- <https://principlesofchaos.org/>
- <https://sre.google/sre-book/testing-reliability/>