Hi all I ve encountered a prefect 2 0 cloud problem I ve a s

Question

Hi all I ve encountered a prefect 2 0 cloud problem I ve a simple a flow which has a single task which looks as follows ``` task name= get subscriptions retries=2 retry delay seconds=5 def get subscriptions paper code logger Logger response = requests get my url return response``` The request itself works fine if I run it manually However as soon as I use prefect 2 0 with prefect 2 0 cloud to run the flow task I run into to following exception The get request in the task takes about 1 minute and 10 seconds to return The exception itself is not coming from the server or my client gt I changed my request get call in the task to a http client request but still get the request exception below so I ve the strong feeling it s somehow related to prefect Exception summary requests exceptions ConnectionError Connection aborted timeout The write operation timed out followed by 10 36 55 875 | ERROR | Flow run chocolate starling Crash detected Request to <https api beta prefect io api accounts bd169b15 9cf0 41df 9e46 2233ca3fcfba workspaces f507fe51 4c9f 400d 8861 ccfaf33b13e4 task runs 29d89dc3 4d92 4c69 a143 44f164303819 set state> timed out Exception details See in thread Is there something wrong in how I use the requests module Or is there a hidden timeout for prefect when a prefect scheduled task runs for more than 1 minute Edit I run the flow currently only locally by running python name of script py Edit2 I m running the python env in WSL2 Edit3 I use GCS storage as my default storage Maybe this causes the problem Edit4 I was able to work around the issue by zipping the content of the response before returning it in my flow So if I change my flow to the following it works For me it looks really as if the upload to GCS has a timeout of 1 minute and therefore the whole flow breaks if the upload takes longer than this minute I can live with this workaround for the moment however I d be happy to know if my theory about GCS being the problem is correct ``` task name= get subscriptions retries=2 retry delay seconds=5 def get subscriptions paper code logger Logger response = requests get my url return zlib compress response content ```

Andreas Nigg · Answer

```Traceback most recent call last File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect engine py line 520 in orchestrate flow run result = await run sync in interruptible worker thread File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect utilities asyncio py line 116 in run sync in interruptible worker thread tg start soon File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio backends asyncio py line 662 in aexit raise exceptions 0 File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio backends asyncio py line 867 in run result = context run func args File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect utilities asyncio py line 96 in capture worker thread and result result = fn args kwargs File home andreas github datawarehouse prefect data integrations kayak kayak integration flows py line 122 in integrate kayak subscriptions print subs response result File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect futures py line 209 in result return sync File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect utilities asyncio py line 221 in sync return run async from worker thread async fn args kwargs File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect utilities asyncio py line 136 in run async from worker thread return anyio from thread run call File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio backends asyncio py line 970 in run async from thread return f result File home andreas miniconda3 envs python prefect test lib python3 9 concurrent futures base py line 446 in result return self get result File home andreas miniconda3 envs python prefect test lib python3 9 concurrent futures base py line 391 in get result raise self exception File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect futures py line 220 in result return final state result raise on failure=raise on failure File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect orion schemas states py line 136 in result raise data File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect task runners py line 212 in submit result = await run fn run kwargs File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect engine py line 793 in begin task run return await orchestrate task run File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect engine py line 927 in orchestrate task run await client persist data File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect client py line 1538 in persist data storage token = await block write data File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect blocks storage py line 309 in write await run sync in worker thread upload File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect utilities asyncio py line 56 in run sync in worker thread return await anyio to thread run sync call cancellable=True File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio backends asyncio py line 867 in run result = context run func args File home andreas miniconda3 envs python prefect test lib python3 9 site packages google cloud storage blob py line 2844 in upload from string self upload from file File home andreas miniconda3 envs python prefect test lib python3 9 site packages google cloud storage blob py line 2571 in upload from file created json = self do upload File home andreas miniconda3 envs python prefect test lib python3 9 site packages google cloud storage blob py line 2388 in do upload response = self do resumable upload File home andreas miniconda3 envs python prefect test lib python3 9 site packages google cloud storage blob py line 2232 in do resumable upload response = upload transmit next chunk transport timeout=timeout File home andreas miniconda3 envs python prefect test lib python3 9 site packages google resumable media requests upload py line 515 in transmit next chunk return request helpers wait and retry File home andreas miniconda3 envs python prefect test lib python3 9 site packages google resumable media requests request helpers py line 171 in wait and retry raise error File home andreas miniconda3 envs python prefect test lib python3 9 site packages google resumable media requests request helpers py line 148 in wait and retry response = func File home andreas miniconda3 envs python prefect test lib python3 9 site packages google resumable media requests upload py line 507 in retriable request result = transport request File home andreas miniconda3 envs python prefect test lib python3 9 site packages google auth transport requests py line 549 in request response = super AuthorizedSession self request File home andreas miniconda3 envs python prefect test lib python3 9 site packages requests sessions py line 542 in request resp = self send prep send kwargs File home andreas miniconda3 envs python prefect test lib python3 9 site packages requests sessions py line 655 in send r = adapter send request kwargs File home andreas miniconda3 envs python prefect test lib python3 9 site packages requests adapters py line 498 in send raise ConnectionError err request=request requests exceptions ConnectionError Connection aborted timeout The write operation timed out ```

Andreas Nigg · Answer

``` most recent call last File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpcore backends asyncio py line 49 in write await self stream send item=buffer File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio streams tls py line 202 in send await self call sslobject method self ssl object write item File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio streams tls py line 168 in call sslobject method await self transport stream send self write bio read File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio backends asyncio py line 1307 in send await self protocol write event wait File home andreas miniconda3 envs python prefect test lib python3 9 asyncio locks py line 226 in wait await fut asyncio exceptions CancelledError During handling of the above exception another exception occurred Traceback most recent call last File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpcore exceptions py line 8 in map exceptions yield File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpcore backends asyncio py line 49 in write await self stream send item=buffer File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio core tasks py line 118 in exit raise TimeoutError TimeoutError During handling of the above exception another exception occurred Traceback most recent call last File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpx transports default py line 60 in map httpcore exceptions yield File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpx transports default py line 353 in handle async request resp = await self pool handle async request req File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpcore async connection pool py line 253 in handle async request raise exc File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpcore async connection pool py line 237 in handle async request response = await connection handle async request request File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpcore async connection py line 90 in handle async request return await self connection handle async request request File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpcore async http11 py line 105 in handle async request raise exc File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpcore async http11 py line 75 in handle async request await self send request body kwargs File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpcore async http11 py line 128 in send request body await self send event event timeout=timeout File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpcore async http11 py line 137 in send event await self network stream write bytes to send timeout=timeout File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpcore backends asyncio py line 49 in write await self stream send item=buffer File home andreas miniconda3 envs python prefect test lib python3 9 contextlib py line 137 in exit self gen throw typ value traceback File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpcore exceptions py line 12 in map exceptions raise to exc exc httpcore WriteTimeout The above exception was the direct cause of the following exception Traceback most recent call last File home andreas github datawarehouse prefect data integrations kayak kayak integration flows py line 211 in lt module gt integrate kayak subscriptions File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect flows py line 370 in call return enter flow run engine from flow call self parameters File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect engine py line 128 in enter flow run engine from flow call return anyio run begin run File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio core eventloop py line 70 in run return asynclib run func args backend options File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio backends asyncio py line 292 in run return native run wrapper debug=debug File home andreas miniconda3 envs python prefect test lib python3 9 asyncio runners py line 44 in run return loop run until complete main File home andreas miniconda3 envs python prefect test lib python3 9 asyncio base events py line 647 in run until complete return future result File home andreas miniconda3 envs python prefect test lib python3 9 site packages anyio backends asyncio py line 287 in wrapper return await func args File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect client py line 107 in with injected client return await fn args kwargs File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect engine py line 192 in create then begin flow run return await begin flow run File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect engine py line 314 in begin flow run terminal state = await orchestrate flow run File home andreas miniconda3 envs python prefect test lib python3 9 contextlib py line 670 in aexit raise exc details 1 File home andreas miniconda3 envs python prefect test lib python3 9 contextlib py line 653 in aexit cb suppress = await cb exc details File home andreas miniconda3 envs python prefect test lib python3 9 contextlib py line 199 in aexit await self gen athrow typ value traceback File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect engine py line 1023 in report flow run crashes await client set flow run state File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect client py line 1642 in set flow run state response = await <http self client post|self client post> File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpx client py line 1842 in post return await self request File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpx client py line 1527 in request return await self send request auth=auth follow redirects=follow redirects File home andreas miniconda3 envs python prefect test lib python3 9 site packages prefect client py line 258 in send await super send args kwargs File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpx client py line 1614 in send response = await self send handling auth File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpx client py line 1642 in send handling auth response = await self send handling redirects File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpx client py line 1679 in send handling redirects response = await self send single request request File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpx client py line 1716 in send single request response = await transport handle async request request File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpx transports default py line 353 in handle async request resp = await self pool handle async request req File home andreas miniconda3 envs python prefect test lib python3 9 contextlib py line 137 in exit self gen throw typ value traceback File home andreas miniconda3 envs python prefect test lib python3 9 site packages httpx transports default py line 77 in map httpcore exceptions raise mapped exc message from exc httpx WriteTimeout```

Anna Geller · Answer

gt I d be happy to know if my theory about GCS being the problem is correct I believe your theory is correct Currently the default mode of handling task run results is to store those as cloudpickled files in the default storage I believe we may improve the process of what happens when the results cannot be stored due to issues connecting to remote storage Thanks for reporting that and nice work finding a workaround I ll open an issue to investigate a more robust approach < Marvin> open Improve timeout logic for handling issues when task run results cannot be written to a remote storage

Marvin · Answer