Hi Prefect community! I have a question about orchestrating workflows using Prefect when there are steps within the workflow that have different infrastructure requirements. For example if I have a workflow I would like to orchestrate using Prefect that has these steps

(DBT job on Snowflake) -> (ML training job).

It would be great to provision a small Fargate task with a Docker image with just the DBT project for the first one, and then provision an ECS task with significantly larger resources and a different Docker image for the ML training job. Is there a way to do this by creating dependencies between Prefect deployments or by specifying infrastructure at the task or sub-flow level?

I highly recommend checking out

run_deployment

(imported like:

from prefect.deployments import run_deployment

). You can deploy flows that need their own infrastructure, like docker images, and then run those deployments from another flow.

Awesome thank you! I'll take a look at that now 🙂