it's because coding agents are getting so good

changing for the better. And it's because coding agents are getting so good. They're getting smarter, they're getting faster. Instagram has scaled to 2 billion users on Postgres. Reddit uses Postgres, Notion runs on Postgres, Heroku, Strava, Discord, they all use Postgres at some capacity. And the reason that all of these systems chose to stay on a database that might not be able to scale is not because they didn't know about the alternatives. They actually knew about the alternatives and they evaluated them, but they still chose to stay with Postgres. And this video will try to answer the question, why do they all still use Postgres? And what does it actually take to make Postgres work at 10th scale? Now, as you can see, I've got a bunch of diagrams to show you guys, and it'll be a highly visual video. And by the end of this video, you should have a clear picture of what the actual bottlenecks are when you scale a relational database. So I want you to stick around till the end. Okay, so a quick bit of history first. It's 2010, Instagram has just launched. It's a very tiny team with just three engineers. And their setup is honestly almost embarrassing in how simple it is. They're running on EC2, which is just Amazon's rented virtual machines. And the photos go into S3 and everything else like the user accounts, photo metadata, comments, likes, the follow graph, all of it goes into a single Postgres database. That's it, that's the whole architecture. Three engineers, one database, no microservices, no Kafka, no fancy infrastructure of any kind, just Postgres doing its job. By 2011, there are 10 million users, but their infra is still simple. By 2012, when Facebook buys them for a billion dollars, there are 27 million users on a two terabyte Postgres database, still with only a few engineers, and that's wild. And in this whole thing, there's a lesson buried that we're going to come back to at the end of this video. And the lesson is boring technology with good engineering beats new technology every single time. But by 2012, the single database setup is starting to crack. Two terabytes is approaching the memory limit of the biggest EC2 instance you could rent at that time, and disk IO is saturated. They've vertically scaled the absolute max, and there's nowhere left to go on on one single machine. So now we are at an important question. What do you do when one machine is not big enough anymore? The wrong answer is to panic and switch databases. The right answer is to figure out exactly why the database is struggling, and then go ahead and fix that. So let's look at what exactly broke first. So the first wall that Instagram hit had nothing to do with data size. It had nothing to do with disk space. It had nothing to do with how many photos that people were uploading. And it was something very simple that most engineers don't even think about, connections. About 1.3 megabytes per connection, and it doesn't sound like much, but Instagram's Django servers were creating new connections constantly. Every Django process kept its own little pool of connections to the database. So imagine you've got 50 application servers and each one keeping 30 connections open to Postgres. That's 1,500 connections. And at 1.3 megabytes each, you're at roughly two gigabytes of memory. And the database hasn't done a single useful thing yet. It's just sitting there, holding connections, eating RAM. The fix is connection pooling, and specifically a tool called PG Bouncer. PG Bouncer is a really lightweight proxy. It sits in between your application and Postgres. And from the application's point of view, it thinks that it's talking to a database, but it's actually talking to PG Bouncer. And PG Bouncer takes all those incoming connections from your apps and multiplexes them onto a much smaller pool of real connections to the actual Postgres server. So instead of 1,500 connections eating two gigabytes of RAM, you've got just 30 real connections handling all of the traffic. And now the database's memory is free for the stuff it actually should be doing, which is caching pages, sorting query results, planning queries, basically the way it's supposed to do. And that's it. Super simple. It's just a proxy. But here's the thing. This is the first invisible bottleneck pretty much every Postgres deployment hits. And most teams don't add a connection pooler until they're already in pain. So if you're running Postgres right now at any kind of meaningful scale, and you don't have PG Bouncer or something similar to it sitting in front of a database, then that is the single highest leverage thing that you can fix this week. Okay, so the.