Tweet by fatih

October 22, 2025

I was a huge advocate for having e2e/integration tests tied to Pull Requests, but I no longer believe that's a great idea. e2e tests by nature are very finicky. They are long and have complex interactions. On top of it, they almost never run with the same stack that Prod is running (you can argue that you have prod similar environments, but nope, it'll be never the same). What happens with time is, because of the underlying assumptions and long tests, the tests start getting flakey. A 3-4 min CI run could end up being 30 min in a few years, and you might think that's the norm. It's not. So what is a better solution? Separate CI and CD. Merge changes only with unit tests, or better tests that passes under 2 minutes. Have a `async` e2e CI pipeline that runs on your main/master branch. Deployment to production is only allowed when the e2e passes. The benefits are immense: * It increases velocity. * Happy engineers * Increases product ship rate * Still have e2e but it no longer blocks you. You might think it's postponing and is the same. It's not. Imagine you're building a car. With the previous approach, everytime you make a change in your car, you need to drive it several miles. Even for small changes. With the later, engineers can focus on shipping the car, add feature, bug fixes etc..., without blocking people working on the car. At @PlanetScale we have multiple repositories and products, there is no one rule. But we have a dedicated 7/24 hour e2e lanes that runs constantly, always. If the e2e lanes are unhappy, we're looking into it. One of the e2e lanes always creates databases, and we have a randomizer that constantly changes certain database parameters. It's basically a self-feeding control loop. That allows us to check if rollouts are working as intended, and if the he database is always up and running. It allows us to catch things that are not possible because with e2e that are created from scratch on each Pull Request, because in the e2e pipeline, the database is never deleted, and it always carries state from the previous rollout. We don't purge these, unless we do major e2e test changes. Of course we also have e2e lanes that do different things, but just wanted to show howe use it.

Author
fatih
Date
October 22, 2025
Open on X