Google Cloud previews Fault Injection Testing for SQL failover and traffic faults

Google Cloud announced Fault Injection Testing in public preview on August 27. The launch post describes reusable experiment templates for two scenarios: triggering failover of a highly available Cloud SQL instance and adding latency or HTTP errors through an Application Load Balancer.

Failure experiments get a managed entry point

The preview includes a read-only dry run that checks permissions and identifies affected resources before the user starts injection. Experiments also have a defined duration, after which the injected faults are reverted.

These controls provide a useful framework for rehearsing disruptions. They do not establish that an application will recover correctly, or that reverting an injected condition undoes every downstream effect. That remains the purpose of the experiment.

For example, a database failover may reveal stale connections, excessive retries, or a transaction that the client cannot confidently classify as committed or failed. An application can recover connectivity while still mishandling the user’s operation.

Decide what success means beforehand

Start with one customer-visible journey and write down the expected degraded behavior. Set a stop condition tied to user impact, alongside an operator who can halt the exercise and an independent way to observe it.

Compare the experiment’s resource list with the dependencies used by that journey. A carefully limited fault can still affect shared components, especially if retrying clients create load elsewhere.

The first valuable result may be a defect in the recovery process rather than a successful resilience demonstration. Record the actual sequence, fix the failure, and repeat the same bounded experiment. Preview tooling is best adopted as a way to produce concrete evidence about recovery, not as a replacement for a complete reliability strategy.