Matthew Perry

04/17/2020, 11:09 PM
Hi everyone 👋 Data engineer with a remote sensing/ IOT sensor background here. I'm looking at Prefect to orchestrate processing-intensive scientific computing workflows on AWS. Currently, we're using AWS Batch + ECS to run docker containers but it's a really limited map-reduce thing. Ideally we'd like a system that would allow us to leverage our existing docker containers and build them into more complex, parameterized DAG topologies. We've tried Airflow... and hit almost all of the limitations mentioned in prefect's docs 🙄 We're just getting started with Prefect but it looks very promising so far! Are there any other folks using prefect to tie together already-dockerized processes?
👀 1


04/17/2020, 11:30 PM
Welcome @Matthew Perry! I know there are a few folks here who have done something similar, so hopefully some of them see your question. In the meantime, depending on your needs, you may find inspiration in our Docker or Kubernetes task libraries - I don’t know that we include an ECS task out of the box but it’s certainly interesting. Let us know if you need any help!

Javier Garcia

08/26/2020, 5:23 PM
@Matthew Perry @Jeremiah - we are trying to orchestrate AWS Batch jobs using Prefect and noticed there are still no pre-cooked Prefect Tasks for managing anything related AWS Batch. Have you heard of any new developments on this front?
There aren't any ECS-related Tasks in the task library either. It seems like something that should already exist. Before we go and implement our own custom Task, I was wondering if you know something I don't about why these are still missing from the Task Library.

Matthew Perry

08/26/2020, 10:17 PM
@Javier Garcia I believe there is a
that takes a
argument which is effectively a self-managed ECS cluster. I haven't tried it but it seems doable.