I struggled with "microservices" until I learned these 22 patterns
#164: A field guide to splitting systems, keeping data correct, and surviving failure in production
Share this letter & I’ll send you some rewards for the referrals.
Some of these microservice patterns are foundational, and some are quite advanced. ALL of them are super useful to software engineers building scalable systems.
Curious to know how many were new to you:
API Gateway,
Database per Service,
Circuit Breaker,
Event-Driven Messaging,
Service Discovery,
Retry with Backoff,
Decompose by Business Capability,
Saga,
Health Check API,
Distributed Tracing,
Service Instance per Container,
API Composition,
Transactional Outbox,
Remote Procedure Invocation (REST/gRPC),
Backends for Frontends (BFF),
Bulkhead,
Idempotent Consumer,
Strangler Fig,
CQRS,
Sidecar/Service Mesh,
Anti-Corruption Layer,
Event Sourcing.
For each, I’ll share:
What it is in plain English & how it works,
A real-world analogy (if I found one),
Tradeoffs,
Why it matters.
Let’s go!
§
The most dangerous app in your company was built by someone in finance last Tuesday (Partner)
The number of vibe-coded apps in enterprise environments has skyrocketed.
Yet most vibe-coded apps cannot handle customer data securely, comply with your internal workflows, or pass production processes.
i.e., they “break” your company standards.
So your apps become an IT problem when you ask yourself these questions:
Where does the data live & who can access the app?
Where are the audit logs & how to enforce company security policies?
Can this safely reach production?
INTRODUCING SUPERBLOCKS…
(Instead of cleaning up later, it makes IT the foundation from day one.)
Apps deploy inside your own AWS account and VPC, so sensitive data stays under your control.
Company login, role-based permissions, audit logs, and governance get enforced automatically across all apps.
Every deployment gets scanned for vulnerabilities before it reaches production.
So you build production-ready apps instead of demos that need to be rebuilt later…and your business moves faster.
If you’re serious about enterprise AI app development, try Superblocks right now:
(Thanks to Superblocks for partnering on this post.)
§
1. API Gateway
An application programming interface (API) gateway is a single entry point for your services.
A client must “never” call services directly.
The gateway gives them a single address and forwards each request to the correct service. Plus, it centralizes the cross-cutting concerns: authenticates requests, enforces rate limits, terminates Transport Layer Security (TLS), caches responses, and translates protocols.
i.e., your services stay focused on business logic instead of repeating cross-cutting logic across the codebase.
Analogy
A hotel front desk routes guest requests.
You never walk into the kitchen or the laundry room yourself. Instead, you ask the desk, and they route your request to the right department.
i.e., one point of contact hides a dozen moving parts.
Tradeoff
A gateway simplifies clients1, centralizes security and observability, and lets you reshape services behind a stable interface.
But it adds a network hop, and could become a single point of failure if you run only one instance. Plus, it can become a development bottleneck when every team must change it, and it risks becoming a mini-monolith if you add business logic inside.
So run it behind a load balancer for high availability and keep it thin.
Why it matters
Use an API gateway when many services sit behind one public API, mobile and web apps need a single endpoint, or you want to handle authentication and rate limits in one place.
It can also help you move clients from a monolith to microservices without breaking the existing API…
Some popular implementations are Kong, NGINX, Envoy, AWS API Gateway, and Apigee.
TIP: Pair it with the Backends for Frontends2 pattern (more on this in concept 15) when different clients, such as web and mobile apps, need different APIs.
2. Database per Service
Each service has a private database that NO other service directly touches.
A service keeps its own data and exposes data only through its API.
Other services ask for what they need instead of directly reaching into the tables. This keeps services loosely coupled: you can change a schema, switch between databases, or add an index, and nothing breaks…
i.e., database becomes an implementation detail hidden behind the service boundary.
Analogy
A shop with its own stockroom that only its staff enter.
Customers ask at the counter, and the staff fetches what’s needed. Nobody wanders into the back.
Because one team controls the stockroom, it stays organized, and changes surprise nobody.
Tradeoff
Private databases give each team independence, isolate failures, and let each service pick the right storage for its job.
But queries that span services get harder, you lose cross-table joins & foreign keys, and keeping data consistent across services needs patterns such as Saga, API Composition, and Transactional Outbox3.
A shared database looks easier on day one but silently couples every service over time.
Why it matters
Every microservices system has independent deploys, or services with very different data shapes & access patterns.
This pattern makes microservices independent. So treat a “shared database” as a red flag.
Popular data stores are PostgreSQL, MySQL, MongoDB, and DynamoDB.
3. Circuit Breaker
A circuit breaker temporarily stops calls to a failing service, giving it time to recover and protecting callers from cascading failures.
The breaker wraps a remote call & watch failures:
When calls succeed, it stays closed and traffic flows.
When failures cross a threshold, it opens, and every call fails fast without touching the failed service.
After a cool-down, it goes half-open and lets one trial call through.
If this succeeds, it closes again; if not, it opens & waits longer.
This turns a slow, resource-draining timeout into an instant, cheap rejection.
Analogy
An electrical fuse trips when too much current flows.
Instead of letting a fault burn down the house, the fuse cuts the circuit. Once the problem is fixed, you reset it and power returns.
The breaker does the same for a failing dependency, cutting it off before it drags everything down.
Tradeoff
They add error-handling paths and need careful tuning of thresholds & timeouts.
Plus, they can trip on a “brief” blip and reject good traffic, and require monitoring so you notice an open circuit.
Why it matters
Calls to a remote/third-party dependency must degrade gracefully, or one slow service must NOT take down the whole request path.
Resilience4j (Java), Polly (.NET), and Envoy proxy implement breakers, and a service mesh (more on this in concept 20) such as Istio can apply them without code changes.
4. Event-Driven Messaging
Services communicate by publishing & consuming events through a “message broker” instead of calling each other directly & wait.
A direct call couples the caller to the callee in time: the caller blocks for a reply, and a failure on either side breaks the request.
Event-driven messaging breaks this link. A service publishes an event to a broker and moves on, and any number of consumers can subscribe & react on their own schedule.
This publish-and-subscribe style also lets you add new consumers later without touching the producer.
Analogy
A phone call versus a group channel:
A phone call needs both people free at the same instant. A message posted to a group channel waits until each reader is ready, and everyone interested sees it.
New people can join the group channel later and still get the updates.
Tradeoff
It trades simple request-reply for eventual consistency.
Plus, debugging gets harder across an event chain, and there’s a need to handle duplicate & out-of-order delivery.
Why it matters
Workflows where steps can happen asynchronously, fanning one event out to many services, decoupling producers from consumers, or smoothing traffic spikes with a buffer.
Apache Kafka, RabbitMQ, AWS SNS and SQS, and Google Pub/Sub are popular broker implementations.
TIP: Pair it with the Idempotent Consumer pattern4 (more on this in concept 17), because at-least-once delivery means duplicates will happen.
5. Service Discovery
Service discovery lets services find each other by name through a registry, which updates itself as instances start, stop & move.
In microservices architecture, instances come and go, and their addresses change constantly. i.e., hard-coding a location breaks the moment a service moves.
Each instance registers its address with a service registry when it starts and de-registers when it stops. Callers query the registry by name to get a current, healthy address.
Here are 2 ways to implement service discovery:
client-side, where the caller queries the registry and load-balances itself,
server-side, where a load balancer or the platform does it.
Analogy
A contacts app instead of numbers scribbled in pen…
When a friend changes their number, the app updates and you can still reach them. The registry keeps every service’s current contact details, so callers always reach them.
Tradeoff
It adds infrastructure,
Makes the registry a critical dependency,
Needs health checks to stay accurate, and a stale cached lookup can still send traffic to a dead instance.
Why it matters
Fixed addresses are NOT practical for auto-scaling fleets, containerized deployments with shifting IPs, blue-green and rolling deploys5.
Consul, etcd, and Netflix’s Eureka are classic registries, while Kubernetes builds discovery in through DNS and Services.
Most service meshes6 include it, so you often get discovery for free at scale.
6. Retry with Backoff
Retry with backoff reattempts a failed call a few times with a growing delay, turning brief glitches into successes without hammering a struggling service.
Networks blip, and a call that fails once often succeeds on the next try.
A naive retry loop can stampede a service that’s already down. Backoff fixes this by waiting longer between each attempt, usually “doubling” the delay, and jitter adds randomness so many clients don’t retry in lockstep.
A retry limit and a timeout cap the total effort so a stuck call gives up instead of hanging forever.
Analogy
Redialing a busy phone line.
You wait a moment and try again, not forty times in a row, and you give up after a few rings rather than holding the line all day.
Each wait is a little longer so you’re not jamming the line.
Tradeoff
Retrying a non-idempotent call can double-charge or double-write; aggressive retries can amplify an outage into a retry storm, and extra attempts raise tail latency7 for the request.
TIP: Only retry idempotent operations, always add jitter, and pair retries with a circuit breaker.
Why it matters
Calls to flaky networks/third-party APIs, transient database errors, or any remote operation that’s safe to repeat.
Resilience4j & Polly provide retry with exponential backoff and jitter out of the box.
Timeout, retry with backoff, and circuit breaker together are standard resilience mechanisms for remote calls.
7. Decompose by Business Capability
Split a system into services along the natural lines of the business, using business capabilities and domain-driven design8 to draw the boundaries.
The HARD part of microservices is deciding where one service ends and the next begins…
Here are two methods to guide you:
Decomposing by business capability splits the system by what the organization does, such as orders, payments, or shipping. Then gives each capability a service.
Decomposing by subdomain uses domain-driven design (DDD) to model the business and carve it into subdomains.
Each becomes a bounded context: a boundary inside which every term carries one precise meaning.
Boundaries drawn this way follow the business, which shifts far more slowly than the code.
Analogy
Think of a company with different departments:
Sales handles selling, Shipping handles deliveries, and Payments handles billing. Each department owns its own work. Even the same word can mean different things. For example, a "customer" means an order and delivery address to Shipping, but payment details & invoices to Payments.
A bounded context keeps each meaning separate, so teams don't interfere with each other.
Tradeoff
Choosing the right service boundaries needs a good understanding of the business.
If you split services the wrong way, they end up constantly calling each other. While changing those boundaries later becomes difficult because each service already owns its own data and responsibilities.
Why it matters
Use this pattern when you’re building a new microservices system, breaking a monolith into smaller services, or giving each team ownership of a specific business area.
§
Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.
When you upgrade, you’ll get:
High-level architecture of real-world systems.
Deep dive into how popular real-world systems work.
How real-world systems handle scale, reliability, and performance.
Ready for the best part?
§
Keep reading with a 7-day free trial
Subscribe to The System Design Newsletter to keep reading this post and get 7 days of free access to the full post archives.