Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

I'm trying to setup a flow usings ECS serverless push workers. Is there any way to have multiple docker images be used in the flow on a per-task basis? I have different docker images for each step because of conflicting dependencies/requirements. I tried asking Marvin, but I think it is hallucinating answers.

Edit: I just found this: <https://discourse.prefect.io/t/is-it-possible-to-direct-specific-tasks-or-flows-to-a-specific-work-pool-or-queue/3077> That is not really viable for me. I really don't want to. have 10 different flows and only be able to run via deploy. Any other suggestions?

It looks like my best solution if I indeed want to use Prefect is to make a task that submits them to ECS and monitors it within the task. I would have to handle all the monitoring if it is done, transferring data etc. within the job submitted to ECS.

Also just found this as well: <https://discourse.prefect.io/t/can-prefect-run-each-task-in-a-different-docker-container/434> I'm still not convinced it is a great solution.

Ok, this method might be a bit viable. Still feels weird to need to deploy each one of them. <https://discourse.prefect.io/t/run-each-task-in-a-different-docker-container-using-prefect-2/2430>