Hello all I ve noticed the following error intermittently us

Question

Hello all I ve noticed the following error intermittently using Prefect Cloud for the last few weeks I m running my agent on an EC2 instance My deployments use docker containers I updated recently from 2 1 0 to 2 3 2 The same errors have occurred with both versions Any insight into why this may be happening ```prefect exceptions PrefectHTTPStatusError Server error 500 Internal Server Error for url Response exception message Internal Server Error For more information check ```

Ross Teach · Answer

stacktrace from the agent logs ```Traceback most recent call last File usr local lib python3 10 site packages prefect engine py line 595 in orchestrate flow run result = await run sync flow call File usr local lib python3 10 site packages prefect utilities asyncutils py line 57 in run sync in worker thread return await anyio to thread run sync call cancellable=True File usr local lib python3 10 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File usr local lib python3 10 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File usr local lib python3 10 site packages anyio backends asyncio py line 867 in run result = context run func args File opt prefect hub prefect flows xandr dbt final flow py line 38 in main is ready = is run dbt ready File opt prefect hub prefect flows xandr dbt final flow py line 76 in is run dbt ready is all ready = all is run dbt ready for day day=day for day in days File opt prefect hub prefect flows xandr dbt final flow py line 76 in lt genexpr gt is all ready = all is run dbt ready for day day=day for day in days File opt prefect hub prefect flows xandr dbt final flow py line 84 in is run dbt ready for day is all valid = all report values valid for day File opt prefect hub prefect flows xandr dbt final flow py line 84 in lt genexpr gt is all valid = all report values valid for day File opt prefect hub prefect flows xandr dbt final flow py line 104 in report values valid for day bigquery rollup column value = get bigquery column value File src dependency injector cwiring pyx line 28 in dependency injector cwiring get sync patched patched File usr local lib python3 10 site packages prefect tasks py line 295 in call return enter task run engine File usr local lib python3 10 site packages prefect engine py line 735 in enter task run engine return run async from worker thread begin run File usr local lib python3 10 site packages prefect utilities asyncutils py line 137 in run async from worker thread return anyio from thread run call File usr local lib python3 10 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File usr local lib python3 10 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 10 concurrent futures base py line 458 in result return self get result File usr local lib python3 10 concurrent futures base py line 403 in get result raise self exception File usr local lib python3 10 site packages prefect engine py line 863 in get task call return value return await future result File usr local lib python3 10 site packages prefect futures py line 236 in result return final state result raise on failure=raise on failure File usr local lib python3 10 site packages prefect orion schemas states py line 143 in result raise data File usr local lib python3 10 site packages prefect task runners py line 203 in submit result = await call File usr local lib python3 10 site packages prefect engine py line 1084 in begin task run return await orchestrate task run File usr local lib python3 10 site packages prefect engine py line 1223 in orchestrate task run state = await propose state File usr local lib python3 10 site packages prefect engine py line 1435 in propose state response = await client set task run state File usr local lib python3 10 site packages prefect client py line 1797 in set task run state response = await <http self client post|self client post> File usr local lib python3 10 site packages httpx client py line 1842 in post return await self request File usr local lib python3 10 site packages httpx client py line 1527 in request return await self send request auth=auth follow redirects=follow redirects File usr local lib python3 10 site packages prefect client py line 279 in send response raise for status File usr local lib python3 10 site packages prefect client py line 225 in raise for status raise PrefectHTTPStatusError from httpx error exc from exc cause ```

Ross Teach · Answer

I do have code where tasks are not called directly from flows I m wondering if the retry mechanism could be failing there ```from prefect import flow from prefect import task flow def flow python method def python method task task retries=3 retry delay seconds=30 def task print HELLO if name == main flow ```

Ross Teach · Answer

Actually I was able to test in local that the following code retried successfully So my previous comment was likely not the issue ```from prefect import flow from prefect import task flow def flow python method def python method task task retries=10 retry delay seconds=1 def task raise Exception error if name == main flow ```

Bianca Hoch · Answer

Hi Ross thanks for reaching out I noticed that you messaged us previously about a <https prefect community slack com archives CL09KU1K7 p1662124883125299|similar issue> If possible can you update to the latest version 2 4 0 and let us know if this continues If the error continues to happen can you tell us a bit more about your deployment

Ross Teach · Answer

Thanks Bianca I will give it a try Yeah I updated recently to 2 3 2 and hoped that might be the issue I will try 2 4 Any insight into why this happening and or why upgrading might help

Robin Weiß · Answer

Hey < Bianca Hoch> I am currently facing exactly the same issues and unfortunately it s blocking our company s decision to go forward with Prefect as the main workflow automation tool I did already update to 2 4 0 and it didn t help Weirdly enough these errors pop up for me after the flows were peacefully running for a few hours It really does seem to be an issue with the Prefect Cloud API from my side but probably it s a layer 8 problem anyway

Anna Geller · Answer

gt it s blocking our company s decision to go forward with Prefect in that case I d highly recommend getting in touch with paid support at <mailto cs prefect io|cs prefect io> where you can get personalized help gt I did already update to 2 4 0 and it didn t help I d recommend sharing your exact steps and minimal reproducible example as a GitHub issue we can try to reproduce could be a bug we can t know without an example and exact details of your setup

Bianca Hoch · Answer

Hi team just wanted to give a quick update that there is an internal ticket opened for this

Ross Teach · Answer

Thank you all I will try different deployment configurations and see if the issue still occurs

Robin Weiß · Answer

< Anna Geller> I am already in contact with < Christopher Boyd> about these issues and he is looking into it thanks slightly smiling face And regarding contacting cs The problem here is that my client would like to get things up and running in a minimalistic setup without paid support We are not even doing any super heavy lifting yet so it would be great to get things done on our own Thanks for your support

Christopher Boyd · Answer

Unfortunately we have seen more of these errors and are investigating the performance

Stefan · Answer

Hi everyone just to add to the above We re also intermittently receiving 500 Internal Server errors The latest one happened after a flow ran successfully for 9 minutes having already completed 50+ task runs Flow run ID is d75a3838 5a33 45e1 b11c 43fe147e1317 and the error `prefect exceptions PrefectHTTPStatusError Server error 500 Internal Server Error for url <https api prefect cloud api accounts workspaces task runs > ` in case this helps with the debugging Happy to provide any additional information you require but unfortunately it is not reproducible so I can t give any specific code example

Stefan · Answer

Still facing random 500er errors which even lead to the agent crashing ```Traceback most recent call last File usr local lib python3 8 site packages prefect cli utilities py line 41 in wrapper return fn args kwargs File usr local lib python3 8 site packages prefect utilities asyncutils py line 212 in wrapper return run async in new loop async fn args kwargs File usr local lib python3 8 site packages prefect utilities asyncutils py line 141 in run async in new loop return anyio run partial fn args kwargs File usr local lib python3 8 site packages anyio core eventloop py line 70 in run return asynclib run func args backend options File usr local lib python3 8 site packages anyio backends asyncio py line 292 in run return native run wrapper debug=debug File usr local lib python3 8 asyncio runners py line 44 in run return loop run until complete main File usr local lib python3 8 asyncio base events py line 616 in run until complete return future result File usr local lib python3 8 site packages anyio backends asyncio py line 287 in wrapper return await func args File usr local lib python3 8 site packages prefect cli agent py line 126 in start await critical service loop File usr local lib python3 8 site packages prefect utilities services py line 39 in critical service loop await workload File usr local lib python3 8 site packages prefect agent py line 118 in get and submit flow runs async for work queue in self get work queues File usr local lib python3 8 site packages prefect agent py line 82 in get work queues work queue = await self client read work queue by name name File usr local lib python3 8 site packages prefect client orion py line 669 in read work queue by name response = await self client get f work queues name name File usr local lib python3 8 site packages httpx client py line 1751 in get return await self request File usr local lib python3 8 site packages httpx client py line 1527 in request return await self send request auth=auth follow redirects=follow redirects File usr local lib python3 8 site packages prefect client base py line 182 in send response raise for status File usr local lib python3 8 site packages prefect client base py line 125 in raise for status raise PrefectHTTPStatusError from httpx error exc from exc cause prefect exceptions PrefectHTTPStatusError Server error 500 Internal Server Error for url <https api prefect cloud api accounts work queues name queue main> ```

Robin Weiß · Answer

I still have the same issues too < Christopher Boyd> is there any updates or an ETA yet

Robin Weiß · Answer

< Stefan> Have you tried reducing the amount of logging messages during flow execution It s just a really random guess from my side but maybe the underlying problem on Prefect s side is somehow related to their logging servers not responding in time Either way it s starting to become a blocker for us and we really hoped this would be fixed or at least addressed somehow by now disappointed

Christopher Boyd · Answer

This seems like a different error than the original ones we were seeing Many of the reported ones were in task run task statte while this appears in the work queue I can take this feedback back to the team There have been improvements made however and pushed so generally speaking this should be much less an issue if you are having issues then traces like this are helpful to isolate