How would horizontal partition work by storing different rows in different tables? Did they write code to do that? Or is it something postgres offer? How does read work if data has to joined or filtered or sorted?
The proxy server is a Golang service. It's where the partitioning logic is implemented, as I understood it. (Postgres seems to have native support for partitioning, along with some extensions available.)
They use the 'scatter-gather pattern' if the data is spread across many databases.
So many databases will be queried in parallel, and the result is combined at the app layer.
There's a performance tradeoff by querying many databases, but it happens rarely with Figma's query patterns.
Hi Neo Kim in one of your previous post I have seen you refrence 3 books for being good at system design . One is designing data intensive application by Martin kelppmann I'm somehow unable to find that post can you please share me the name of other two. I been trying since a week thank you
Here is a lesson on how different it is to scale read than write.
Thanks for sharing, Neo.
thanks Raul
There are some very important tips like caching, separation of concerns, and dealing with heavy load.
Great breakdown, Neo! 🙌
How would horizontal partition work by storing different rows in different tables? Did they write code to do that? Or is it something postgres offer? How does read work if data has to joined or filtered or sorted?
The proxy server is a Golang service. It's where the partitioning logic is implemented, as I understood it. (Postgres seems to have native support for partitioning, along with some extensions available.)
They use the 'scatter-gather pattern' if the data is spread across many databases.
So many databases will be queried in parallel, and the result is combined at the app layer.
There's a performance tradeoff by querying many databases, but it happens rarely with Figma's query patterns.
As long as it is designed in such a way that requires fewer joins then performance won't be a bottleneck :)
Did they do manual sharding in code or sharding using citus which is extension on postgres?
Hi Neo Kim in one of your previous post I have seen you refrence 3 books for being good at system design . One is designing data intensive application by Martin kelppmann I'm somehow unable to find that post can you please share me the name of other two. I been trying since a week thank you
This is a great article about how to scale PostgreSQL, the most popular relational database. Thank you, Neo!