I'm trying to setup a flow usings ECS serverless push workers. Is there any way to have multiple docker images be used in the flow on a per-task basis? I have different docker images for each step because of conflicting dependencies/requirements. I tried asking Marvin, but I think it is hallucinating answers.
Edit: I just found this: https://discourse.prefect.io/t/is-it-possible-to-direct-specific-tasks-or-flows-to-a-specific-work-pool-or-queue/3077 That is not really viable for me. I really don't want to. have 10 different flows and only be able to run via deploy. Any other suggestions?
Ben
08/12/2024, 8:10 PM
It looks like my best solution if I indeed want to use Prefect is to make a task that submits them to ECS and monitors it within the task. I would have to handle all the monitoring if it is done, transferring data etc. within the job submitted to ECS.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.