Hey on prefect 2 cloud I randomly encounter internal server

Question

Hey on prefect 2 cloud I randomly encounter internal server error s when the agent wants to communicate with the server exception see in thread I have a flow which spawns up to 50 Tasks the tasks are all done within seconds I run the flow currently every 5 minutes Am I running in some sort of api limitation due to too many tasks Is there something to prevent this internal server error from happening This error happens approx every 15th time all other times the flow runs just fine ```File usr local lib python3 9 site packages prefect client py line 226 in raise for status raise PrefectHTTPStatusError from httpx error exc from exc cause prefect exceptions PrefectHTTPStatusError Server error 500 Internal Server Error for url <https api prefect cloud api accounts bd169b15 9cf0 41df 9e46 2233ca3fcfba workspaces f507fe51 4c9f 400d 8861 ccfaf33b13e4 task runs |https api prefect cloud api accounts bd169b15 9cf0 41df 9e46 2233ca3fcfba workspaces f507fe51 4c9f 400d 8861 ccfaf33b13e4 task runs > Response exception message Internal Server Error ```

Andreas Nigg · Answer

```Encountered exception during execution Traceback most recent call last File usr local lib python3 9 site packages prefect engine py line 587 in orchestrate flow run result = await run sync flow call File usr local lib python3 9 site packages prefect utilities asyncutils py line 116 in run sync in interruptible worker thread tg start soon File usr local lib python3 9 site packages anyio backends asyncio py line 662 in aexit raise exceptions 0 File usr local lib python3 9 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File usr local lib python3 9 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File usr local lib python3 9 site packages anyio backends asyncio py line 867 in run result = context run func args File usr local lib python3 9 site packages prefect utilities asyncutils py line 96 in capture worker thread and result result = fn args kwargs File teeevents flow py line 203 in save nonblacklist leads to salesforce update synced events submit bigquery dataset name bigquery table name File usr local lib python3 9 site packages prefect tasks py line 491 in submit return enter task run engine File usr local lib python3 9 site packages prefect engine py line 727 in enter task run engine return run async from worker thread begin run File usr local lib python3 9 site packages prefect utilities asyncutils py line 136 in run async from worker thread return anyio from thread run call File usr local lib python3 9 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File usr local lib python3 9 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 9 concurrent futures base py line 446 in result return self get result File usr local lib python3 9 concurrent futures base py line 391 in get result raise self exception File usr local lib python3 9 site packages prefect engine py line 837 in create task run then submit task run = await create task run File usr local lib python3 9 site packages prefect engine py line 877 in create task run task run = await flow run context client create task run File usr local lib python3 9 site packages prefect client py line 1693 in create task run response = await File usr local lib python3 9 site packages httpx client py line 1842 in post return await self request File usr local lib python3 9 site packages httpx client py line 1527 in request return await self send request auth=auth follow redirects=follow redirects File usr local lib python3 9 site packages prefect client py line 280 in send response raise for status File usr local lib python3 9 site packages prefect client py line 226 in raise for status raise PrefectHTTPStatusError from httpx error exc from exc cause prefect exceptions PrefectHTTPStatusError Server error 500 Internal Server Error for url Response exception message Internal Server Error ```

Zanie · Answer

Hi We re getting a team together to investigate and resolve these errors Have you set up retries on your flow run

Andreas Nigg · Answer

Hey yes I use retries 2 retries But the flow immediately enters failed state after this exception occurs without attempting to retry And it only ever occurs for this rather high number of tasks flow For other flows i most of the time have only max 10 tasks or so there it never occurs It might be reproducible with spawning a lot of tasks and running the flow in a rather short interval Thanks a lot for investigating

Andreas Nigg · Answer

Do you want me to create a github issue

Zanie · Answer

There are a couple existing ones but I think we re lacking a simple MRE If you can make one of those that d be really helpful slightly smiling face

Andreas Nigg · Answer

Hey Michael I think I can try to create an MRE if an MRE is an easy to reproduce example sweat smile sweat smile sweat smile Just to be sure What are actually API Task limits with the cloud solution I think I will simply try to create a flow with a good amount of tasks and similar task structure as my real world flow But this might mean I could create quite a good amount of api requests to prefect not that my monthly quota or so gets hit by trying to reproduce the isse angel

Zanie · Answer

There are no limits right now

Zanie · Answer

We enforce rate limits but the client will retry when encountered

Andreas Nigg · Answer

Hey Michael I was able to recreate a different issue with a simple example I encountered this one also with the same flow as described above so maybe they are related The issue is When having quite a good amount of tasks the flow execution sometimes just stops No failing no crash no exception If you run the following sample flow it stops somewhere but never continues Is there something I m missing or is this interesting for you guys With stopping I simply mean that the log outputs stop at some point without finishing all the tasks and not continuing I waited for about 10 minutes before coming to the conclusion that the flow might be stuck slightly smiling face I ran the flow locally prefect 2 4 5 python myfilename py ```from time import sleep from prefect import flow task get run logger from prefect task runners import ConcurrentTaskRunner import requests flow name= my favorite function description= Such description task runner=ConcurrentTaskRunner timeout seconds=1000 def my favorite function logger = get run logger This is my hopefully failing flow test = call api submit 0 logger test1 = call api submit 0 logger test2 = call api submit 0 logger ctr = 0 while True ctr += 1 if ctr gt = 200 break useless nr = call api submit ctr logger another api submit ctr useless nr logger return task name= call api description= Such wow retries=2 retry delay seconds=10 def call api url ctr logger ++++++ hello austrian news site +++++++ + str ctr response = requests get url timeout=30 sleep 2 response str ctr return response task name= another api description= Such another wow retries=2 retry delay seconds=10 def another api url ctr useless nr logger another + str ctr response = requests get url timeout=30 sleep 2 I m so useless + str useless nr status code response str ctr return response if name == main my favorite function ```