Hello I have a really weird error When I run 3 5 flows in th

Question

Hello I have a really weird error When I run 3 5 flows in the same moment there isn t any problem But when I try to run over 50 flows I get the error `concurrent futures base CancelledError` I see that my cluster hasn t any problem to create over 50 pods simultaneously Queue concurrency limit is 500 Stack trace in thread PS Does it rely that during transition phase to Prefect2 I used free cloud account if yes where can I can see limits

Aleksandr Liadov · Answer

```Encountered exception during execution Traceback most recent call last File opt pysetup venv lib python3 8 site packages prefect engine py line 637 in orchestrate flow run result = await run sync flow call File opt pysetup venv lib python3 8 site packages prefect utilities asyncutils py line 91 in run sync in worker thread return await anyio to thread run sync File opt pysetup venv lib python3 8 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File opt pysetup venv lib python3 8 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File opt pysetup venv lib python3 8 site packages anyio backends asyncio py line 867 in run result = context run func args File app flows 2 main flow py line 129 in main flow features execution flow results = add task runner to features execution flow File opt pysetup venv lib python3 8 site packages prefect flows py line 448 in call return enter flow run engine from flow call File opt pysetup venv lib python3 8 site packages prefect engine py line 168 in enter flow run engine from flow call return run async from worker thread begin run File opt pysetup venv lib python3 8 site packages prefect utilities asyncutils py line 177 in run async from worker thread return anyio from thread run call File opt pysetup venv lib python3 8 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File opt pysetup venv lib python3 8 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr lib python3 8 concurrent futures base py line 442 in result raise CancelledError concurrent futures base CancelledError```

Aleksandr Liadov · Answer

```Crash detected Execution was interrupted by an unexpected exception Traceback most recent call last File usr lib python3 8 contextlib py line 189 in aexit await self gen athrow typ value traceback File opt pysetup venv lib python3 8 site packages prefect task runners py line 163 in start yield self File opt pysetup venv lib python3 8 site packages prefect engine py line 372 in begin flow run terminal or paused state = await orchestrate flow run File opt pysetup venv lib python3 8 site packages prefect engine py line 637 in orchestrate flow run result = await run sync flow call File opt pysetup venv lib python3 8 site packages prefect utilities asyncutils py line 91 in run sync in worker thread return await anyio to thread run sync File opt pysetup venv lib python3 8 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File opt pysetup venv lib python3 8 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future asyncio exceptions CancelledError During handling of the above exception another exception occurred concurrent futures base CancelledError objects blacklist 03eab60f 0 ccdd4886963f49fda76ea528fb1fcd96 1```

Walter Cavinaw · Answer

Just out of curiosity where is it running Gke aws azure This log is from the parent flow creating 50 flows

Aleksandr Liadov · Answer

It s running on Gke log from parent flow which runs 2 subflows So 50 flows run = 50 3 150 flows are executing in the same moment

Aleksandr Liadov · Answer

One of subflows is running on Dask cluster

Aleksandr Liadov · Answer

< Christopher Boyd> < Ryan Peden> maybe you have an idea what I should see

Walter Cavinaw · Answer

Are you running on autopilot or standard cluster

Walter Cavinaw · Answer

I m just asking because Crashed usually means it s the infrastructure or failed network Orion calls If you are running on GKE Autopilot it could be because pods are getting evicted to scale down the nodes a few other people have had this issue It might even happen with a regular GKE cluster but I m not sure

Aleksandr Liadov · Answer

< Walter Cavinaw> thanks for your questions We have a regular gke cluster The problem here it seems to me from dask But I observe it only if i try to run more than 10 flows