Tim Galvin

08/20/2023, 11:02 AM
A question to the prefect community. Is there a way where certain to execute certain task runners on a task runner other than the one specified by the main flow? My use case is pretty simple. A lot of my tasks require many cores, but some long running tar + g jobs only need a single core. I would like to try to specific a separate dask task runner that only uses a single core per work, rather than the dask task runner using 32 per worker. I have previously looked into the insides of a
class before, and see that there is a mechanism to specify a task runner - this internal option is what denoted the blocking vs non-blocking behaviour when directly calling a task vs calling a tasks
method. For instance: seems to suggest I could extend either the
method to pass through a different task runner. Would this work?