@Marvin In a Prefect 3 flow, can each task of the flow run on different Docker Images ?

In Prefect 3.x, it's currently not officially documented or supported to run each task within a flow on different Docker images. This is a feature that many users would find beneficial, especially for tasks that have varying resource requirements, such as ETL processes or machine learning tasks that require different dependencies or computing power. There is a GitHub issue (#15246) discussing the desire for heterogeneous task compute resources, which includes running tasks on different Docker images. This feature isn't implemented yet but is proposed for future updates. For now, you might need to adopt a workaround by manually configuring your environment or using different work queues and workers for tasks that need different Docker images. You can keep an eye on Prefect's GitHub and their release notes for updates on when this feature might be available. For more details, check out the related GitHub issue discussion here.

there's 2 common ways of doing this • define a deployment that should run in a container and then call it from a task

@task
def do_work_in_container(**kwargs):
  run_deployment(**kwargs)

• just talk to docker directly from python

@task
def do_work_in_container(**kwargs):
  docker_client = ...
  docker_client.create(...) etc

Hi Nate, thank you so much for the quick response. Could you elaborate a bit on the first part ? Do you mean to create a flow run from an existing deployment ?( and then treat it like an API almost ? ) It seems like there are so many incredible things you can do with Prefect, but my main painpoint is that there's so little tutorials. I wish Prefect can take off soon so the community can drive this.

main painpoint is that there's so little tutorials

I hear you! working on it - one of the challenges is that this is very true

It seems like there are so many incredible things you can do with Prefect

and so covering all the surface area of tutorials can be hard!

Could you elaborate a bit on the first part ? Do you mean to create a flow run from an existing deployment ?( and then treat it like an API almost ? )

yes! somewhere else you

prefect deploy

(yaml) or

my_flow.deploy(...)

(python) so Prefect knows about the deployment and where it should run then you can trigger it at any time from the UI, CLI or even other flows with

run_deployment