Hi everyone! Ben here. I fear that I have let another 18 months go by without posting anything on this blog. I swear I do enjoy writing, but apparently only when I have things to write about?
It turns out I do have something to write about! I won’t bury the lede or anything. Earlier today, I locked in a three-year compute cost rate for Grundo’s Cafe. This gives us discounted computing as long as we lock in the expectation that we will continue buying it. Like us or not, we are here until fall 2029! This probably does sound like a strange thing to celebrate, so I’ll back up a bit to contextualize this momentous occasion.
A few months ago, I had a blast taking Habitarium to testing, and then to release. This was a big personal milestone for me. Not just because I am a bug lover through and through (I just did a quick check, 15 of the last 20 photos on my phone’s camera reel are various butterflies, dragonflies, and robber flies). But also because I had been working on Habitarium before Grundo’s Cafe even existed!
And okay well, I wasn’t working on it for four years straight. More accurately, I would work on it a little and then get distracted by something. Then work on it a little more, then get distracted by another thing. Then work on it even more, then get completely overwhelmed by extracting dozens of corrupted P3s poses from 15 year old SWFs. And decide silly things like “creating an entire plot sounds easier than figuring out how to make a Pinchit look like it is carrying a water droplet without stuttering”.
I had Habitarium ready for testing in late May 2026. At around the same time, the site was facing some of the worst lag it has seen. I’ve talked about the many-headed hydra that is site lag here before. But something different about this instance is that we were starting to scrape the bottom of the barrel for code fixes to the lag. There were no obvious and easy targets for improvement left. Each n+1 query we could find and amend was a very marginal benefit. Each web page we could rewrite into a single-page application represented just a small bite of the whole enchilada. A real fix would need to be a resource fix, and those kinds of fixes cost money.
Since the site’s inception, we had used Heroku for our server provisioning. Let me tell you - when I was learning web development in the late 2010s, Heroku was a godsend. You could write an application and provision a web server and a database entirely for free. Free! I had made many toy games and tiny projects using these free resources, and I learned a lot from doing so. As I learned later, there is no such thing as a free lunch. The price we were paying was being beholden to their ecosystem. When Heroku got rid of its free tier in fall 2022, hobby devs everywhere cried out. Me included. But inertia kept us in place.
The value proposition of Heroku made sense to me, too. Beyond the free tier, you are paying for them to manage your infrastructure for you. Provisioning more server compute and a beefier database was as straightforward as sliding a slider on a dashboard. In the earlier days of the site, this seemed reasonable. But this past May, staring down a $150/mo database upgrade as the only possible improvement tier…. It felt untenable.
In terms of alternatives, there are generally two options. On-premises self-hosting, and cloud provisioning the resources. Regrettably, self-hosting in the age of AI is a nightmare. Hardware prices are incredibly turbulent and seem to be constantly ballooning with no end in sight (at least we get Fruit Love Island out of it though). Furthermore, the initial capital cost would be at least thousands of dollars and operationally it would be a headache. So cloud it is.
In terms of cloud providers, there are some options here like AWS, GCP, and Azure. AWS was the obvious choice for a few reasons. Heroku to AWS is a well-documented migration path. The team was generally the most familiar with AWS. And AWS is generally considered the most widely-used stack for the kind of development we’re doing.
So talk of abandoning Heroku altogether got serious in early June. Being newly freed from the shackles of Habi development (I was still fixing bugs that were coming out of UAT but this was lower-intensity), I was happy to have another thing to focus on. That being said, I can’t give enough credit to devs like Rodolfo and CrystalFlame, who were churning out Terraform like it was no one’s business.
By June 15th, we were ready to execute phase one of the migration plan: web app cutover. If you want to impress your nerd friends, you can also call this a blue-green deployment. Basically, we instantiated the new web app on AWS and run it in parallel to the old web app, and then cutover traffic from the old to the new. We collectively held our breath and.. It worked! For a while.
Shot + chaser... That September 20th downtime event was because credential rotation snuck up on us. Woops!
There were a few more issues to sort out. Remember that the entire premise of this migration was cost savings so that we could combat lag by provisioning more compute resources. Well, a few days into phase 1, we noticed a $200 ghost in the machine. The blue-green web server deployment was good because the database migration is very risky in comparison. It is where we can risk losing data, years of account progress, etc. So we resolved to leave DATABASE_URL pointed at Heroku Postgres for a bit while we made sure the app layer on AWS was solid.
That “for a bit” turned out to be an expensive bit. Every single database query, every page load, every timed task, every websocket event that touched the DB - they all were leaving our private AWS network, routing out through a NAT gateway, crossing the public internet to Heroku’s Postgres, and coming all the way back. We were doing this constantly, to the tune of about 6 GB of traffic every hour. This round trip was costing us roughly $217 a month.
Moving the database turned out to be an interesting problem anyways. Postgres restores aren’t a single dumb copy. If you load tables in the wrong order, foreign key constraints blow the entire thing up. So we split it into three passes: schema first (no constraints yet). Then the data itself (which, on 383 tables, is quite hefty). Then finally all the constraints and indexes once every row was safely in place. All of it tunneled through an SSM port-forward session to a live ECS task since the RDS instance has no public endpoint at all. Imagine the entirety of Grundo's Cafe data is in a water tank on a truck (the data backup), and I am trying to move it into a swimming pool in my backyard (the new RDS instance). The SSM port-forward session running on my laptop was like a drinking straw that the water has to pass through.
By the end of this four hour downtime, I was yelling at my computer "please let me go to bed!"
Once we did this, we could finally point DATABASE_URL at the RDS we provisioned on AWS instead of Heroku. This fixed our NAT gateway costs in an instant. Hooray!
But back to the original point, and the cause for writing this whole blog post. The past three months of running Grundo’s Cafe on AWS has been mostly smooth! It has involved quite a lot of tuning. Database instances, worker node counts, autoscaling parameters. All of these are knobs that require dialing. Now that we’ve dialed the knobs in and they seem mostly stable, we can take advantage of savings plans! These are less mysterious than they sound. AWS is essentially trading a discount for certainty. When someone pays on-demand, AWS has to keep spare capacity around for anyone who might spin up a server with no notice. That flexibility is expensive for them to provide, and the consumer is paying for it.
A savings plan flips that. We’re telling AWS, in writing, exactly how much compute we plan to use and for how long, which lets them plan hardware capacity, power, and cooling around a predictable schedule instead of guessing. The longer the commitment, the more of that guesswork is removed from their side. A three year commitment allows for upwards of a 40% cost savings on compute (I will update the site FAQ monthly expenses soon), which I am eager to reinvest into the never-ending battle against lag.
Of course, the tradeoff is that the risk moves to us. I don’t know what I am going to have for breakfast a week from now, let alone the shape of Grundo’s Cafe in 2029. But both of those unknowns are risks I am okay with. Plus, three year spans have been known to go by in a blink. That is only... two blog posts from now! (kidding (unless))
So if you get one thing from this blog post, know this - despite the up’s and down’s of this site and the smaller day-to-day fires, I am interested in seeing where GC goes for the long haul. And as always, thank you for being interested in the journey as well! It means more than words can express!