In 15 minutes, Barstool Sports saved millions in outage avoidance

In 2017 Andrew Barba was hired as an iOS engineer for Barstool. When he found himself in charge of the engineering team, he knew he wanted to choose an attractive stack for engineers. He knew Node.js and AWS would be the backbone, but didn’t want to burden his team with having to be DevOps experts. The team quickly built a pretty great serverless stack leveraging Lambda, REST APIs, and iOS Native apps, and wanted a database that would play nice with Lambda. Amazon’s Aurora database (cloud-hosted MySQL) seemed to be a logical fit.

The problem they noticed is a common one with serverless architectures. Andrew explains this as a mismatch of velocity. While their Lambda processes could scale incredibly fast, adding thousands of containers per second, their attempts to add read replicas to support new Lambda clients were extremely slow and expensive. “We’d just start adding read replicas, but this would take 3-4 minutes. This is the difference from scaling front-end to back-end. Most databases, including Aurora, just aren’t able to handle the demands of a serverless product. You are limited by how powerful your database is.”

Solving these issues is doable in-house, but Andrew knew it would require a lot more time and indeed a lot more engineers. This was at odds with how he wanted his engineering team to operate. So, he began shopping for a database and looked at many serverless cloud-native options. The team’s requirements were:

The team at Barstool signed up quickly after finding PlanetScale, and in September of 2021, created a trial database, and started syncing data from their main Aurora database to their PlanetScale database. At the time this was a fairly manual process, but it’s now much easier with PlanetScale’s Import feature. Because Barstool had so many outages, they decided to run data through PlanetScale concurrently alongside Aurora to be able to monitor how it handled their traffic. As Andrew describes it, the Aurora outages were incredibly costly, with their worst, a 45-minute outage quickly losing them a couple of million dollars. During one of their events, and during the PlanetScale trial, a query started misfiring and they had another complete Aurora outage. As Andrew describes it — that was the moment they switched to PlanetScale.

Because PlanetScale is running MySQL, Barstool was able to switch very quickly by merging a single branch of code that only contained a new connection string and a few driver changes.

“MySQL went down completely. A query wasn’t executing. We could see that the PlanetScale database was handling it fine on a newer version of MySQL. We literally cut over in 15 minutes. We did a big post-mortem and five months later we haven’t looked back. PlanetScale is incredibly powerful and we just let it do its thing.”

What the team at Barstool had not anticipated was seeing their bill go down after switching over to PlanetScale. “Honestly we did not anticipate the savings from a time or money perspective,” said Andrew “There were unplanned costs, like migrations, when managing our own database. We would have to scale up our resources to 4x to do a single migration. PlanetScale just handles migrations with a click of a button. In the end, we saved 20-30% by switching to PlanetScale.” Since PlanetScale prices its cloud database on usage instead of resources, scaling up and down has no effect on cost. Instead,the database automatically adjusts resources, behind the scenes, as needed for the usage.

In the end, Barstool says the added benefits of working with PlanetScale just keep coming. When asked, Andrew said the team at PlanetScale has made his team’s task much simpler.

“We used to check the AWS dashboard practically nightly. Honestly, we never think about PlanetScale. That’s the way it should be. The reality is that our team is called the product team — we build products. We don’t want to be DevOps experts, and frankly we don’t need DevOps database work. We want to always focus on making our product better.”