Hi Prefect community! I have a question about orc...
# best-practices
Hi Prefect community! I have a question about orchestrating workflows using Prefect when there are steps within the workflow that have different infrastructure requirements. For example if I have a workflow I would like to orchestrate using Prefect that has these steps
(DBT job on Snowflake) -> (ML training job).
It would be great to provision a small Fargate task with a Docker image with just the DBT project for the first one, and then provision an ECS task with significantly larger resources and a different Docker image for the ML training job. Is there a way to do this by creating dependencies between Prefect deployments or by specifying infrastructure at the task or sub-flow level?
Also for more context, I am very new to Prefect and have been trying it out using ECSTasks with the integrated storage + infrastructure pattern on the Docker images.
I highly recommend checking out
(imported like:
from prefect.deployments import run_deployment
). You can deploy flows that need their own infrastructure, like docker images, and then run those deployments from another flow.
gratitude thank you 1
🚀 1
🙌 1
Awesome thank you! I'll take a look at that now 🙂