Hi everyone 👋 Data engineer with a remote sensing/ IOT sensor background here. I'm looking at Prefect to orchestrate processing-intensive scientific computing workflows on AWS. Currently, we're using AWS Batch + ECS to run docker containers but it's a really limited map-reduce thing. Ideally we'd like a system that would allow us to leverage our existing docker containers and build them into more complex, parameterized DAG topologies. We've tried Airflow... and hit almost all of the limitations mentioned in prefect's docs 🙄 We're just getting started with Prefect but it looks very promising so far! Are there any other folks using prefect to tie together already-dockerized processes?

Welcome @Matthew Perry! I know there are a few folks here who have done something similar, so hopefully some of them see your question. In the meantime, depending on your needs, you may find inspiration in our Docker or Kubernetes task libraries - I don’t know that we include an ECS task out of the box but it’s certainly interesting. Let us know if you need any help!

@Matthew Perry @Jeremiah - we are trying to orchestrate AWS Batch jobs using Prefect and noticed there are still no pre-cooked Prefect Tasks for managing anything related AWS Batch. Have you heard of any new developments on this front?

@Javier Garcia I believe there is a

FargateTaskEnvironment

that takes a

launch_type='ec2'

argument which is effectively a self-managed ECS cluster. I haven't tried it but it seems doable.