What is the best method to split task resources up...
# prefect-community
n
What is the best method to split task resources up in a Flow. For example if I want a task to connect to a large spark cluster to do preprocessing, while for my ML task I want it to get connected to a GPU instance.
k
Subflows is one because the executor is on a subflow level. Resource manager is also another option. Dask annotations is another possibility. From the DaskExecutor docs:
Copy code
Note that if you have tasks with tags of the form "dask-resource:KEY=NUM" they will be parsed and passed as Worker Resources of the form {"KEY": float(NUM)} to the Dask Scheduler.
👍 1
n
TY ill take a look 😄
a
also, if you want to run each task in a different container/Kubernetes pod to handle resource allocation this way, check this Discourse topic