Hi prefect team !! I currently want to run tasks in parallel on prefect-server on k8s. However, when I submit multiple tasks in a

for

statement using Dask, I get the following error, probably because there are too many tasks.

Crash detected! Request to <http://prefect-server.prefect.svc.cluster.local:4200/api/task_runs/> failed: PoolTimeout:.

Generally, how can this be resolved? I would like to consider using subflows as parallel processes, is this the right idea? The documentation seems to recommend Dask, so I should not easily increase the number of flows if there are other causes for parallel processes to fail. Also, if I increase the number of flows, how can I limit the number of concurrent executions?

Hi @tig, did you find a solution for this issue? I get a similar error trying to run parallel tasks on ECS.

@Romain Vincent It was resolved, but I’m sorry, I don’t know exactly why. At the time, I gave up on parallel processing in tasks and tried to parallelise by splitting it into several flows. After a while, I changed it back to task again and it was able to run normally. The problem may have been solved by adding more CPU and memory. … But I’m sorry, I can’t promise that the information will be helpful, because I don’t remember exactly what was going on at the time either.