If you use Prefect on Google Cloud, you might enjoy this post I wrote about how to run Prefect flows as Google Cloud Run jobs: https://medium.com/the-prefect-blog/serverless-prefect-flows-with-google-cloud-run-jobs-23edbf371175

Super interesting post! We currently self-host Prefect on Kubernetes but a lot of our tasks invoke AWS Lambda functions due to the massive reduction in cost for compute. This Cloud Run model seems to be the solution to allow all our code to live solely within Prefect and not be split between two tools. Do you know of a similar solution for AWS users? Seem like Fargate and Lambda don’t quite have all the features needed! Thanks

Hi Fred, we have an infrastructure block that lets you use ECS, which is (more or less) the AWS equivalent. The example code at that link shows how you can specify either a Fargate or EC2 launch type, so that might be a good way to get around the Fargate limitations you mentioned. My colleague wrote an excellent post about using ECS for running Prefect flows. It uses ECS + Fargate, but I believe you could adapt it to use ECS + EC2 fairly easily. She also created a GitHub repository that demonstrates how to set this all up as part of a CI/CD pipeline.

Thanks Ryan, I’ll take a look!