Chaos Engineering for AI Infrastructure: Fault injection and resilience testing for GPUs, model serving, RAG pipelines, and agentic systems on Kubernetes

How long do you think it would take you to notice if your AI system started giving worse answers?

Honestly, for most teams, it’s weeks, and the news comes from a customer. Chaos engineering answers that question for infrastructure in about thirty seconds, because if a server goes down, it either reroutes or it doesn’t. And language models completely shatter all those assumptions. Their output varies for real, their decay builds up instead of happening straight away, and those failures are confident, plausible, and wrong while latency, error rate, and pod health stay exactly where they should be.

This book sets out the new framework for AI infrastructure. You’ll find that, working through one support-assistant application across twelve chapters, everything moves from defining steady state for probabilistic systems to injecting faults at every layer, including accelerators, inference servers, model artefacts, retrieval pipelines, model APIs, autonomous agents, distributed training and the monitoring stackitself.

Just to be clear, the toolchain is intentionally kept small. The LitmusChaosorchestrates, the Chaos Meshreaches the kernel and filesystem, and the mitmproxyhandles the content-level faults that neither can touch, extended with custom injectors where nothing suitable exists. So, every experiment comes to one of these four conclusions, and then the last chapter sums them all up as a programme scorecard.

Key Learnings

• Write down what “working correctly” means when your system’s output legitimately varies.
• Break GPUs and inference servers on purpose, with a restore path that always runs.
• Spot the failures that keep every dashboard green and every alert silent.
• Check model weights for damage before loading them, not after serving starts.
• Catch the model release that scores better overall while getting worse where it matters.
• Find out whether your RAG pipeline is helping, or quietly making answers worse.
• Corrupt what the model returns, then watch whether your agent notices.
• Stop your remediation agent from making an incident worse than it was.
• Prove your training checkpoints actually resume the run they claim to continue.
• Catch slow decay in hours instead of days by changing how you alert.

Table of Content

1. Why AI Systems Fail Differently?
2. Steady State for Probabilistic System
3. Lab and the Toolchain
4. Accelerator and Compute Chaos
5. Inference Serving Chaos
6. Model Artifacts, Registries, and Version Skew
7. Retrieval and Context Chaos
8. Model-Boundary Chaos
9. Agentic and Multi-Agent Chaos
10. Distributed Training and Choas Fine-Tuning
11. Observing Probabilistic Degradation
12. Operationalizing Chaos