Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hey all! Wanted to share my org's story on migrating from Civis (an analyst-friendly but limited orchestration tool) to Prefect. <https://austinweisgrau.github.io/migrating-from-civis-to-prefect.html|Here's my blog post about it.>

I cover the <https://austinweisgrau.github.io/migrating-to-prefect-part-1-civis-woes.html|limitations of Civis> vis a vis data engineering best practices and how Prefect improves our workflows. I cover what it looks like to <https://austinweisgrau.github.io/migrating-to-prefect-part-4-moving-a-script-from-civis-to-prefect.html|migrate a script from Civis to Prefect>.

I get into details about how we <https://austinweisgrau.github.io/migrating-to-prefect-part-2-prefect-aws-ecs.html|set up our execution layer in AWS ECS> based on the <https://towardsdatascience.com/prefect-aws-ecs-fargate-github-actions-make-serverless-dataflows-as-easy-as-py-f6025335effc|dataflowops tutorial >with a few modifications. I include a sample repository with working template scripts. I also demo a modification to Prefect's concurrency implementation to support <https://austinweisgrau.github.io/migrating-to-prefect-part-3-rate-limiting-api-calls.html|rate limiting API calls>.

I always appreciate seeing demos of how folks go about their implementation and hope this is helpful if you're looking into how to set up Prefect in AWS ECS or move off of a limited legacy orchestration stack!

Nice, thanks for sharing! <@U0380MW6MHU> used to work on Civis and also migrated (himself) to Prefect, so I’m sure he’d be excited to hear about this!