Early praise for Release It! Second Edition
Mike is one of the software industry’s deepest thinkers and clearest communica-
tors. As beautifully written as the original, the second edition of Release It! extends
the first with modern techniques—most notably continuous deployment, cloud
infrastructure, and chaos engineering—that will help us all build and operate
large-scale software systems.
➤ Randy Shoup
VP Engineering, Stitch Fix
If you are putting any kind of system into production, this is the single most im-
portant book you should keep by your side. The author’s enormous experience
in the area is captured in an easy-to-read, but still very intense, way. In this up-
dated edition, the new ways of developing, orchestrating, securing, and deploying
real-world services to different fabrics are well explained in the context of the core
resiliency patterns.
➤ Michael Hunger
Director of Developer Relations Engineering, Neo4j, Inc.
So much ground is covered here: patterns and antipatterns for application re-
silience, security, operations, architecture. That breadth would be great in itself,
but there’s tons of depth too. Don’t just read this book—study it.
➤ Colin Jones
CTO at 8th Light and Author of Mastering Clojure Macros
Release It! is required reading for anyone who wants to run software to production
and still sleep at night. It will help you build with confidence and learn to expect
and embrace system failure.
➤ Matthew White
Author of Deliver Audacious Web Apps with Ember 2
I would recommend this book to anyone working on a professional software project.
Given that this edition has been fully updated to cover technologies and topics
that are dealt with daily, I would expect everyone on my team to have a copy of
this book to gain awareness of the breadth of topics that must be accounted for
in modern-day software development.
➤ Andy Keffalas
Software Engineer/Team Lead
A must-read for anyone wanting to build truly robust, scalable systems.
➤ Peter Wood
Software Programmer
Release It! Second Edition
Design and Deploy Production-Ready Software
Michael T. Nygard
The Pragmatic Bookshelf
Raleigh, North Carolina
Many of the designations used by manufacturers and sellers to distinguish their products
are claimed as trademarks. Where those designations appear in this book, and The Pragmatic
Programmers, LLC was aware of a trademark claim, the designations have been printed in
initial capital letters or in all capitals. The Pragmatic Starter Kit, The Pragmatic Programmer,
Pragmatic Programming, Pragmatic Bookshelf, PragProg and the linking g device are trade-
marks of The Pragmatic Programmers, LLC.
Every precaution was taken in the preparation of this book. However, the publisher assumes
no responsibility for errors or omissions, or for damages that may result from the use of
information (including program listings) contained herein.
Our Pragmatic books, screencasts, and audio books can help you and your team create
better software and have more fun. Visit us at https://pragprog.com.
The team that produced this book includes:
Publisher: Andy Hunt
VP of Operations: Janet Furlow
Managing Editor: Brian MacDonald
Supervising Editor: Jacquelyn Carter
Development Editor: Katharine Dvorak
Copy Editor: Molly McBeath
Indexing: Potomac Indexing, LLC
Layout: Gilson Graphics
For sales, volume licensing, and support, please contact support@pragprog.com.
For international rights, please contact rights@pragprog.com.
Copyright © 2018 The Pragmatic Programmers, LLC.
All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted,
in any form, or by any means, electronic, mechanical, photocopying, recording, or otherwise,
without the prior consent of the publisher.
Printed in the United States of America.
ISBN-13: 978-1-68050-239-8
Encoded using the finest acid-free high-entropy binary digits.
Book version: P1.0—January 2018
Contents
Acknowledgments
.
.
.
.
.
.
.
.
.
.
.
xi
Preface .
.
.
.
.
.
.
.
.
.
.
.
.
.
xiii
.
.
.
.
.
.
.
.
.
Living in Production .
.
Aiming for the Right Target
The Scope of the Challenge
A Million Dollars Here, a Million Dollars There
Use the Force
Pragmatic Architecture
Wrapping Up
Part I — Create Stability
.
.
Case Study: The Exception That Grounded an Airline .
The Change Window
The Outage
Consequences
Postmortem
Hunting for Clues
The Smoking Gun
An Ounce of Prevention?
.
.
.
.
.
.
.
.
.
Stabilize Your System .
Defining Stability
Extending Your Life Span
Failure Modes
Stopping Crack Propagation
Chain of Failure
Wrapping Up
1
2
3
3
4
5
6
9
10
12
14
14
16
18
20
23
24
25
26
27
28
30
1.
2.
3.
Contents • vi
4.
5.
6.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Stability Antipatterns
Integration Points
Chain Reactions
Cascading Failures
Users
Blocked Threads
Self-Denial Attacks
Scaling Effects
Unbalanced Capacities
Dogpile
Force Multiplier
Slow Responses
Unbounded Result Sets
Wrapping Up
.
Stability Patterns
Timeouts
Circuit Breaker
Bulkheads
Steady State
Fail Fast
Let It Crash
Handshaking
Test Harnesses
Decoupling Middleware
Shed Load
Create Back Pressure
Governor
Wrapping Up
Part II — Design for Production
Case Study: Phenomenal Cosmic Powers,
.
Itty-Bitty Living Space .
.
.
.
.
.
.
Baby’s First Christmas
Taking the Pulse
Thanksgiving Day
Black Friday
Vital Signs
Diagnostic Tests
31
33
46
49
51
62
69
71
75
78
80
84
86
90
91
91
95
98
101
106
108
111
113
117
119
120
123
125
129
130
131
132
132
134
135