The System Design Newsletter

The System Design Newsletter

Share this post

The System Design Newsletter
The System Design Newsletter
Tumblr Shares Database Migration Strategy With 60+ Billion Rows
Copy link
Facebook
Email
Notes
More
User's avatar
Discover more from The System Design Newsletter
Weekly newsletter to help busy engineers become good at system design
Over 150,000 subscribers
Already have an account? Sign in

Tumblr Shares Database Migration Strategy With 60+ Billion Rows

#3: Read Now - Amazing Database Migration Technique (4 minutes)

Neo Kim's avatar
Neo Kim
Sep 10, 2023
17

Share this post

The System Design Newsletter
The System Design Newsletter
Tumblr Shares Database Migration Strategy With 60+ Billion Rows
Copy link
Facebook
Email
Notes
More
2
Share

Get the powerful template to approach system design for FREE on newsletter sign-up:


Database migration gone wrong could result in data loss, downtime, and compatibility issues.

This post outlines the database migration techniques at Tumblr. If you want to learn more, scroll to the bottom and find the references.

  • Share this post & I'll send you some rewards for the referrals.

Tumblr is a micro-blogging platform that allows users to publish short blog posts.


How to Migrate a MySQL Database?

The fact that Tumblr uses MySQL database to store its critical data is wild.

The MySQL database at Tumblr consumes 21 terabytes with 60+ billion relational rows. And there are over 200 database servers.

Spreading the database leaders across many data centers provides scalability and high availability. So, they decided to migrate the database leader between data centers.

A brute force approach to database migration.

The database leader ran in the remote data center. So, switch off the traffic toward the remote data center. Migrate the database leader from the remote to the local data center. And Route the traffic toward the local data center.

But this is a recipe for poor availability.

Tumblr wanted the user impact to be very minimal on database migration.

CQRS pattern
CQRS pattern

An optimal approach to database migration.

The command and query responsibility segregation (CQRS) pattern simplified their database migration. The CQRS is a pattern that separates read and write operations to a database.

The most common replication topology in MySQL is leader-follower. The database leader accepts read-write requests. The database followers accept read requests and replicate the leader.

Leader-follower replication across data centers
Leader-follower replication across data centers

The database leader ran in the remote data center. And they provisioned database followers in the local data center. The persistent connections routed the read-write requests to the database leader. The connection reuse over a large number of queries allowed to reduce latency.

But each application service had a different usage pattern. This resulted in frequent disconnections to the database leader. The connection establishment with the database was expensive. So, they ran into a latency problem.

Database proxy
Database proxy

As a workaround, they provisioned a database proxy (ProxySQL) in the local data center. The database proxy held persistent connections to the remote database leader.

The database proxy allowed database connection pooling. Database connection pooling is the method of holding database connections open for reuse by others. This approach prevented frequent disconnections and improved performance.

How to migrate a MySQL database
How Tumblr migrates their MySQL database

They wanted to migrate the database leader from remote data center A to B. The workflow of the database migration at Tumblr is as follows:

  1. Each data center stored the metadata of database followers, the leader, and the database proxy

  2. They migrated the database leader from remote data center A to B

  3. An automation tool pointed all followers to the new leader

  4. An automation tool pointed the database proxy to the new leader

The database followers accepted read requests during the database migration. This approach resulted in minimal user impact with seconds of read-only state. The write requests were either rejected or buffered for a few seconds.

How to further improve write availability during database migration?

The leader-leader replication topology in MySQL could improve write availability during database migration. But this approach introduces the risk of data conflicts. This might be the reason why Tumblr doesn’t use it.


👋 PS - Are you unhappy at your current job?

While preparing for system design interviews to get your dream job can be stressful.

Don't worry, I'm working on content to help you pass the system design interview. I'll make it easier - you spend only a few minutes each week to go from 0 to 1. Yet paid subscription fees will be higher than current pledge fees.

So pledge now to get access at a lower price.

“An excellent newsletter to learn system design through practical case studies.” Franco


Consider subscribing to get simplified case studies delivered straight to your inbox:


Author NK; System design case studies
Follow me on LinkedIn | YouTube | Threads | Twitter | Instagram | Bluesky

Thank you for supporting this newsletter. Consider sharing this post with your friends and get rewards. Y’all are the best.

system design newsletter

Share


This Is How Quora Shards MySQL to Handle 13+ Terabytes

This Is How Quora Shards MySQL to Handle 13+ Terabytes

NK
·
September 3, 2023
Read full story
8 Reasons Why WhatsApp Was Able to Support 50 Billion Messages a Day With Only 32 Engineers

8 Reasons Why WhatsApp Was Able to Support 50 Billion Messages a Day With Only 32 Engineers

NK
·
August 27, 2023
Read full story

Word-of-mouth referrals like yours help this community grow - Thank you.

Feedback from a reader
Feedback from a reader

You can share your testimonial by posting on Twitter or LinkedIn. Then send me the link by replying to this email. Or, you can reply to this email with an anonymous testimonial.


References

  • Tumblr Engineering Team. (2016, October 13). Juggling Databases Between Datacenters. Tumblr Engineering Blog.

  • Engineering Team. (2012, June 8). Jetpants: A Toolkit for Huge MySQL Topologies. Tumblr Engineering Blog.

  • Atchison, L. (2019, November 26). Avoiding Downtime When Migrating Data to the Cloud. LinkedIn.

  • Microsoft. CQRS pattern. Microsoft Azure Architecture Patterns.

  • What is database pooling? (n.d.). In Stack Overflow.


Subscribe to The System Design Newsletter

By Neo Kim · Launched 2 years ago
Weekly newsletter to help busy engineers become good at system design
Vaibhav Patil's avatar
ritesh sonawane's avatar
John Sawiris's avatar
Breno Henrique's avatar
Abilash's avatar
17 Likes∙
2 Restacks
17

Share this post

The System Design Newsletter
The System Design Newsletter
Tumblr Shares Database Migration Strategy With 60+ Billion Rows
Copy link
Facebook
Email
Notes
More
2
Share

Discussion about this post

User's avatar
8 Reasons Why WhatsApp Was Able to Support 50 Billion Messages a Day With Only 32 Engineers
#1: Learn More - Awesome WhatsApp Engineering (6 minutes)
Aug 27, 2023 • 
Neo Kim
737

Share this post

The System Design Newsletter
The System Design Newsletter
8 Reasons Why WhatsApp Was Able to Support 50 Billion Messages a Day With Only 32 Engineers
Copy link
Facebook
Email
Notes
More
24
How PayPal Was Able to Support a Billion Transactions per Day With Only 8 Virtual Machines
#30: Learn More - Awesome PayPal Engineering (4 minutes)
Dec 26, 2023 • 
Neo Kim
246

Share this post

The System Design Newsletter
The System Design Newsletter
How PayPal Was Able to Support a Billion Transactions per Day With Only 8 Virtual Machines
Copy link
Facebook
Email
Notes
More
14
How Stripe Prevents Double Payment Using Idempotent API
#45: A Simple Introduction to Idempotent API (4 minutes)
May 9, 2024 • 
Neo Kim
380

Share this post

The System Design Newsletter
The System Design Newsletter
How Stripe Prevents Double Payment Using Idempotent API
Copy link
Facebook
Email
Notes
More
29

Ready for more?

© 2025 Neo Kim
Publisher Privacy
Substack
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture

Share

Copy link
Facebook
Email
Notes
More

Create your profile

User's avatar

Only paid subscribers can comment on this post

Already a paid subscriber? Sign in

Check your email

For your security, we need to re-authenticate you.

Click the link we sent to , or click here to sign in.