Hello, I am using Prefect-dask: DaskTaskrunner for concurrent task processing.
But somehow it always runs 2 task concurrently. Other 5 task are in pending stage. Is it because of the resource of the server or do I need to change any configuration?
Thanks in advance
c
Christopher Boyd
01/12/2023, 3:25 PM
Hi @Mrityunjoy Das - I think this needs a little bit more detail to properly say.
How many resources are available?
Are you using a local dask cluster, or connecting to an existing dask cluster?
what does your dask configuration look like for # workers / threads?
How are you submitting the work to dask (e.g. the actual task code doing the submission)?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.