I'm trying to use priority with the DaskTaskRunner and looking at the docs we should be able to use dask annotations to achieve this. However, this priority example provided in the docs runs sequentially. It seems unnecessary to prioritise work if it runs sequentially. Am I missing something here?
r
Ryan Peden
02/13/2024, 9:06 PM
It looks like the docs mostly just show that you can use Dask annotations with Prefect tasks the same way you can use them with any other Python function. In the examples, you could call
show.submit()
instead of just show if you wanted to let the tasks run concurrently with different priorities.
If you're connecting to an existing Dask cluster, those priority annotations might still be useful even if your tasks run sequentially, because they'll determine the priority your tasks get relative to other work submitted to the cluster.
n
Noah Holm
02/13/2024, 9:39 PM
Makes sense that it might be useful on a shared dask cluster, didn't think of that
Noah Holm
02/13/2024, 9:40 PM
I tried the submit method as well, but didn't manage to solve the prioritisation then. The tasks ran with a non-deterministic order each run in a local cluster
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.