prefect 2 6 6 local server I am using run deployment to run

Question

prefect 2 6 6 local server I am using run deployment to run two flows i e Flow A and Flow B second flow starts after first flow finishes Whenever I triggers the flows from run deployment flow the flow fails with an error although the Flow A completes without an error

Faheem Khan · Answer

Loading flow for deployment combine flows 07 21 11 AM DEBUG Starting ConcurrentTaskRunner submitted tasks will be run concurrently 07 21 12 AM DEBUG Executing flow combine flows for flow run active cormorant 07 21 12 AM DEBUG Executing combine flows 07 21 12 AM ERROR Encountered exception during execution Traceback most recent call last File usr local lib python3 10 asyncio selector events py line 854 in read ready data received data = self sock recv self max size ConnectionResetError Errno 104 Connection reset by peer The above exception was the direct cause of the following exception Traceback most recent call last File usr local lib python3 10 site packages httpcore exceptions py line 8 in map exceptions yield File usr local lib python3 10 site packages httpcore backends asyncio py line 33 in read return await self stream receive max bytes=max bytes File usr local lib python3 10 site packages anyio backends asyncio py line 1274 in receive raise self protocol exception anyio BrokenResourceError During handling of the above exception another exception occurred Traceback most recent call last File usr local lib python3 10 site packages httpx transports default py line 60 in map httpcore exceptions yield File usr local lib python3 10 site packages httpx transports default py line 353 in handle async request resp = await self pool handle async request req File usr local lib python3 10 site packages httpcore async connection pool py line 253 in handle async request raise exc File usr local lib python3 10 site packages httpcore async connection pool py line 237 in handle async request response = await connection handle async request request File usr local lib python3 10 site packages httpcore async connection py line 90 in handle async request return await self connection handle async request request File usr local lib python3 10 site packages httpcore async http11 py line 105 in handle async request raise exc File usr local lib python3 10 site packages httpcore async http11 py line 84 in handle async request = await self receive response headers kwargs File usr local lib python3 10 site packages httpcore async http11 py line 148 in receive response headers event = await self receive event timeout=timeout File usr local lib python3 10 site packages httpcore async http11 py line 177 in receive event data = await self network stream read File usr local lib python3 10 site packages httpcore backends asyncio py line 30 in read with map exceptions exc map File usr local lib python3 10 contextlib py line 153 in exit self gen throw typ value traceback File usr local lib python3 10 site packages httpcore exceptions py line 12 in map exceptions raise to exc exc httpcore ReadError The above exception was the direct cause of the following exception Traceback most recent call last File usr local lib python3 10 site packages prefect engine py line 610 in orchestrate flow run result = await run sync flow call File usr local lib python3 10 site packages prefect utilities asyncutils py line 68 in run sync in worker thread return await <http anyio to|anyio to> thread run sync call cancellable=True File usr local lib python3 10 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File usr local lib python3 10 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File usr local lib python3 10 site packages anyio backends asyncio py line 867 in run result = context run func args File opt prefect src combine flows py line 25 in combine flows run deployment name= Main downloader downloader 0 6 2 File usr local lib python3 10 site packages prefect utilities asyncutils py line 197 in coroutine wrapper return run async from worker thread async fn args kwargs File usr local lib python3 10 site packages prefect utilities asyncutils py line 148 in run async from worker thread return anyio from thread run call File usr local lib python3 10 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File usr local lib python3 10 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 10 concurrent futures base py line 458 in result return self get result File usr local lib python3 10 concurrent futures base py line 403 in get result raise self exception File usr local lib python3 10 site packages prefect client utilities py line 47 in with injected client return await fn args kwargs File usr local lib python3 10 site packages prefect deployments py line 127 in run deployment flow run = await client read flow run flow run id File usr local lib python3 10 site packages prefect client orion py line 1439 in read flow run response = await self client get f flow runs flow run id File usr local lib python3 10 site packages httpx client py line 1751 in get return await self request File usr local lib python3 10 site packages httpx client py line 1527 in request return await self send request auth=auth follow redirects=follow redirects File usr local lib python3 10 site packages prefect client base py line 159 in send await super send args kwargs File usr local lib python3 10 site packages httpx client py line 1614 in send response = await self send handling auth File usr local lib python3 10 site packages httpx client py line 1642 in send handling auth response = await self send handling redirects File usr local lib python3 10 site packages httpx client py line 1679 in send handling redirects response = await self send single request request File usr local lib python3 10 site packages httpx client py line 1716 in send single request response = await transport handle async request request File usr local lib python3 10 site packages httpx transports default py line 352 in handle async request with map httpcore exceptions File usr local lib python3 10 contextlib py line 153 in exit self gen throw typ value traceback File usr local lib python3 10 site packages httpx transports default py line 77 in map httpcore exceptions raise mapped exc message from exc httpx ReadError

Carlo · Answer

We are seeing the same http error reset peer on the new server version Didn t see on 2 6 4

Carlo · Answer

See this regarding syncing dependencies <https prefect community slack com archives CL09KU1K7 p1667226152022199>

Bianca Hoch · Answer

Hi Faheem thank you for raising this I ll reach out to the team to see what could be going on here Would you mind sharing a few example IDs of the the affected flow runs Also would you mind sharing a minimal reproducible example of your code and provide a description of how you created the deployment

Bianca Hoch · Answer

Hello team < Carlo> < Faheem Khan> just a heads up that we may have for what could be happening here

Faheem Khan · Answer

< Bianca Hoch> the error appears in agent logs even if I am not running any flows but it affects running flows below is agent logs no flow running `prefect agent 1 | 23 07 49 438 | DEBUG | prefect agent Checking for flow runs ` prefect agent 1 | 23 07 49 447 | ERROR | prefect agent prefect agent 1 | Traceback most recent call last prefect agent 1 | File usr local lib python3 10 asyncio selector events py line 854 in read ready data received prefect agent 1 | data = self sock recv self max size prefect agent 1 | ConnectionResetError Errno 104 Connection reset by peer prefect agent 1 | prefect agent 1 | The above exception was the direct cause of the following exception prefect agent 1 | prefect agent 1 | Traceback most recent call last prefect agent 1 | File usr local lib python3 10 site packages httpcore exceptions py line 8 in map exceptions prefect agent 1 | yield prefect agent 1 | File usr local lib python3 10 site packages httpcore backends asyncio py line 33 in read prefect agent 1 | return await self stream receive max bytes=max bytes prefect agent 1 | File usr local lib python3 10 site packages anyio backends asyncio py line 1274 in receive prefect agent 1 | raise self protocol exception prefect agent 1 | anyio BrokenResourceError prefect agent 1 | prefect agent 1 | During handling of the above exception another exception occurred prefect agent 1 | prefect agent 1 | Traceback most recent call last prefect agent 1 | File usr local lib python3 10 site packages httpx transports default py line 60 in map httpcore exceptions prefect agent 1 | yield prefect agent 1 | File usr local lib python3 10 site packages httpx transports default py line 353 in handle async request prefect agent 1 | resp = await self pool handle async request req prefect agent 1 | File usr local lib python3 10 site packages httpcore async connection pool py line 253 in handle async request prefect agent 1 | raise exc prefect agent 1 | File usr local lib python3 10 site packages httpcore async connection pool py line 237 in handle async request prefect agent 1 | response = await connection handle async request request prefect agent 1 | File usr local lib python3 10 site packages httpcore async connection py line 90 in handle async request prefect agent 1 | return await self connection handle async request request prefect agent 1 | File usr local lib python3 10 site packages httpcore async http11 py line 105 in handle async request prefect agent 1 | raise exc prefect agent 1 | File usr local lib python3 10 site packages httpcore async http11 py line 84 in handle async request prefect agent 1 | = await self receive response headers kwargs prefect agent 1 | File usr local lib python3 10 site packages httpcore async http11 py line 148 in receive response headers prefect agent 1 | event = await self receive event timeout=timeout prefect agent 1 | File usr local lib python3 10 site packages httpcore async http11 py line 177 in receive event prefect agent 1 | data = await self network stream read prefect agent 1 | File usr local lib python3 10 site packages httpcore backends asyncio py line 30 in read prefect agent 1 | with map exceptions exc map prefect agent 1 | File usr local lib python3 10 contextlib py line 153 in exit prefect agent 1 | self gen throw typ value traceback prefect agent 1 | File usr local lib python3 10 site packages httpcore exceptions py line 12 in map exceptions prefect agent 1 | raise to exc exc prefect agent 1 | httpcore ReadError prefect agent 1 | prefect agent 1 | The above exception was the direct cause of the following exception prefect agent 1 | prefect agent 1 | Traceback most recent call last prefect agent 1 | File usr local lib python3 10 site packages prefect agent py line 154 in get and submit flow runs prefect agent 1 | queue runs = await self client get runs in work queue prefect agent 1 | File usr local lib python3 10 site packages prefect client orion py line 759 in get runs in work queue prefect agent 1 | response = await prefect agent 1 | File usr local lib python3 10 site packages httpx client py line 1842 in post prefect agent 1 | return await self request prefect agent 1 | File usr local lib python3 10 site packages httpx client py line 1527 in request prefect agent 1 | return await self send request auth=auth follow redirects=follow redirects prefect agent 1 | File usr local lib python3 10 site packages prefect client base py line 159 in send prefect agent 1 | await super send args kwargs prefect agent 1 | File usr local lib python3 10 site packages httpx client py line 1614 in send prefect agent 1 | response = await self send handling auth prefect agent 1 | File usr local lib python3 10 site packages httpx client py line 1642 in send handling auth prefect agent 1 | response = await self send handling redirects prefect agent 1 | File usr local lib python3 10 site packages httpx client py line 1679 in send handling redirects prefect agent 1 | response = await self send single request request prefect agent 1 | File usr local lib python3 10 site packages httpx client py line 1716 in send single request prefect agent 1 | response = await transport handle async request request prefect agent 1 | File usr local lib python3 10 site packages httpx transports default py line 352 in handle async request prefect agent 1 | with map httpcore exceptions prefect agent 1 | File usr local lib python3 10 contextlib py line 153 in exit prefect agent 1 | self gen throw typ value traceback prefect agent 1 | File usr local lib python3 10 site packages httpx transports default py line 77 in map httpcore exceptions prefect agent 1 | raise mapped exc message from exc prefect agent 1 | httpx ReadError prefect agent 1 | 23 07 54 465 | DEBUG | prefect agent Checking for flow runs

Faheem Khan · Answer

Then there s another error in agent logs No flows running other error is broken pipe `prefect agent 1 | 23 11 21 748 | DEBUG | prefect agent Checking for flow runs ` prefect agent 1 | 23 11 21 755 | ERROR | prefect agent prefect agent 1 | Traceback most recent call last prefect agent 1 | File usr local lib python3 10 asyncio selector events py line 916 in write prefect agent 1 | n = self sock send data prefect agent 1 | BrokenPipeError Errno 32 Broken pipe prefect agent 1 | prefect agent 1 | The above exception was the direct cause of the following exception prefect agent 1 | prefect agent 1 | Traceback most recent call last prefect agent 1 | File usr local lib python3 10 site packages httpcore exceptions py line 8 in map exceptions prefect agent 1 | yield prefect agent 1 | File usr local lib python3 10 site packages httpcore backends asyncio py line 33 in read prefect agent 1 | return await self stream receive max bytes=max bytes prefect agent 1 | File usr local lib python3 10 site packages anyio backends asyncio py line 1274 in receive prefect agent 1 | raise self protocol exception prefect agent 1 | anyio BrokenResourceError prefect agent 1 | prefect agent 1 | During handling of the above exception another exception occurred prefect agent 1 | prefect agent 1 | Traceback most recent call last prefect agent 1 | File usr local lib python3 10 site packages httpx transports default py line 60 in map httpcore exceptions prefect agent 1 | yield prefect agent 1 | File usr local lib python3 10 site packages httpx transports default py line 353 in handle async request prefect agent 1 | resp = await self pool handle async request req prefect agent 1 | File usr local lib python3 10 site packages httpcore async connection pool py line 253 in handle async request prefect agent 1 | raise exc prefect agent 1 | File usr local lib python3 10 site packages httpcore async connection pool py line 237 in handle async request prefect agent 1 | response = await connection handle async request request prefect agent 1 | File usr local lib python3 10 site packages httpcore async connection py line 90 in handle async request prefect agent 1 | return await self connection handle async request request prefect agent 1 | File usr local lib python3 10 site packages httpcore async http11 py line 105 in handle async request prefect agent 1 | raise exc prefect agent 1 | File usr local lib python3 10 site packages httpcore async http11 py line 84 in handle async request prefect agent 1 | = await self receive response headers kwargs prefect agent 1 | File usr local lib python3 10 site packages httpcore async http11 py line 148 in receive response headers prefect agent 1 | event = await self receive event timeout=timeout prefect agent 1 | File usr local lib python3 10 site packages httpcore async http11 py line 177 in receive event prefect agent 1 | data = await self network stream read prefect agent 1 | File usr local lib python3 10 site packages httpcore backends asyncio py line 30 in read prefect agent 1 | with map exceptions exc map prefect agent 1 | File usr local lib python3 10 contextlib py line 153 in exit prefect agent 1 | self gen throw typ value traceback prefect agent 1 | File usr local lib python3 10 site packages httpcore exceptions py line 12 in map exceptions prefect agent 1 | raise to exc exc prefect agent 1 | httpcore ReadError prefect agent 1 | prefect agent 1 | The above exception was the direct cause of the following exception prefect agent 1 | prefect agent 1 | Traceback most recent call last prefect agent 1 | File usr local lib python3 10 site packages prefect agent py line 154 in get and submit flow runs prefect agent 1 | queue runs = await self client get runs in work queue prefect agent 1 | File usr local lib python3 10 site packages prefect client orion py line 759 in get runs in work queue prefect agent 1 | response = await <http self client post|self client post> prefect agent 1 | File usr local lib python3 10 site packages httpx client py line 1842 in post prefect agent 1 | return await self request prefect agent 1 | File usr local lib python3 10 site packages httpx client py line 1527 in request prefect agent 1 | return await self send request auth=auth follow redirects=follow redirects prefect agent 1 | File usr local lib python3 10 site packages prefect client base py line 159 in send prefect agent 1 | await super send args kwargs prefect agent 1 | File usr local lib python3 10 site packages httpx client py line 1614 in send prefect agent 1 | response = await self send handling auth prefect agent 1 | File usr local lib python3 10 site packages httpx client py line 1642 in send handling auth prefect agent 1 | response = await self send handling redirects prefect agent 1 | File usr local lib python3 10 site packages httpx client py line 1679 in send handling redirects prefect agent 1 | response = await self send single request request prefect agent 1 | File usr local lib python3 10 site packages httpx client py line 1716 in send single request prefect agent 1 | response = await transport handle async request request prefect agent 1 | File usr local lib python3 10 site packages httpx transports default py line 352 in handle async request prefect agent 1 | with map httpcore exceptions prefect agent 1 | File usr local lib python3 10 contextlib py line 153 in exit prefect agent 1 | self gen throw typ value traceback prefect agent 1 | File usr local lib python3 10 site packages httpx transports default py line 77 in map httpcore exceptions prefect agent 1 | raise mapped exc message from exc prefect agent 1 | httpx ReadError

Faheem Khan · Answer

I was using prefect 2 0 4 and everything was working fine

Bianca Hoch · Answer

Hi Faheem just wanted to loop back and say that we are still investigating this on our end Has bumping down to an older version of Prefect resolved this issue for you

Faheem Khan · Answer

Thanks < Bianca Hoch> yeah it works with older versions