Prefect 2 6 has brought me back but I am getting the occasio

Question

Prefect 2 6 has brought me back but I am getting the occasional error like this This looks to be a failure with the Prefect API Can anyone shed some light on it for me Thanks ```11 20 34 559 | ERROR | Task run Get Items d8ed86f1 2199 Crash detected Request to <https api prefect cloud api accounts cafe3a79 624b 468d 87a7 97fde3358a01 workspaces 5d09d677 90b8 4ef8 a9be 9760b422937a task runs a13a62a5 5e54 4b1d a976 98635a6913c6> failed Traceback most recent call last File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages anyio streams tls py line 130 in call sslobject method result = func args File usr lib python3 10 ssl py line 975 in do handshake self sslobj do handshake ssl SSLWantReadError The operation did not complete read ssl c 997 ```

Zanie · Answer

Hm That s the full traceback < Zach Angell> did you see this during the reliability wor

Tim Enders · Answer

I think this is fthe full stack trace ```11 20 55 376 | ERROR | Task run Get Items d8ed86f1 326 Crash detected Request to <https api prefect cloud api accounts cafe3a79 624b 468d 87a7 97fde3358a01 workspaces 5d09d677 90b8 4ef8 a9be 9760b422937a task runs c8013fad e78f 4da0 a9e6 8084f4e1aec6> failed Traceback most recent call last File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages anyio streams tls py line 130 in call sslobject method result = func args File usr lib python3 10 ssl py line 975 in do handshake self sslobj do handshake ssl SSLWantReadError The operation did not complete read ssl c 997 During handling of the above exception another exception occurred Traceback most recent call last File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpcore backends asyncio py line 67 in start tls ssl stream = await anyio streams tls TLSStream wrap File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages anyio streams tls py line 122 in wrap await wrapper call sslobject method ssl object do handshake File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages anyio streams tls py line 137 in call sslobject method data = await self transport stream receive File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages anyio backends asyncio py line 1265 in receive await self protocol read event wait File usr lib python3 10 asyncio locks py line 214 in wait await fut asyncio exceptions CancelledError During handling of the above exception another exception occurred Traceback most recent call last File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpcore exceptions py line 8 in map exceptions yield File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpcore backends asyncio py line 76 in start tls raise exc File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpcore backends asyncio py line 66 in start tls with anyio fail after timeout File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages anyio core tasks py line 118 in exit raise TimeoutError TimeoutError During handling of the above exception another exception occurred Traceback most recent call last File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpx transports default py line 60 in map httpcore exceptions yield File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpx transports default py line 353 in handle async request resp = await self pool handle async request req File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpcore async connection pool py line 253 in handle async request raise exc File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpcore async connection pool py line 237 in handle async request response = await connection handle async request request File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpcore async connection py line 86 in handle async request raise exc File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpcore async connection py line 63 in handle async request stream = await self connect request File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpcore async connection py line 150 in connect stream = await stream start tls kwargs File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpcore backends asyncio py line 64 in start tls with map exceptions exc map File usr lib python3 10 contextlib py line 153 in exit self gen throw typ value traceback File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpcore exceptions py line 12 in map exceptions raise to exc exc httpcore ConnectTimeout The above exception was the direct cause of the following exception httpx ConnectTimeout```

Zanie · Answer

Oh well that s really helpful slightly smiling face It s a connection timeout

Zanie · Answer

You could bump `PREFECT API REQUEST TIMEOUT` I m not sure why you d be experiencing a long connection time though

Tim Enders · Answer

what is the default value I don t currently have it set

Zanie · Answer

``` prefect config view show defaults | grep TIMEOUT PREFECT API REQUEST TIMEOUT= 30 0 from defaults ```

Tim Enders · Answer

TIL cool

Zanie · Answer

slightly smiling face Is your internet particularly slow 30s is a long time and could be an issue on our end

Tim Enders · Answer

let me get some internet data I just upgraded to fiber a few weeks ago though

Tim Enders · Answer

My internet looks great 369 Mbps down 74 up ping of 15ms

Zanie · Answer

Using the default task runner concurrent

Tim Enders · Answer

corrct

Zanie · Answer

Great thanks We ll need to wait for someone that knows more about recently Cloud performance work to chime in Presumably we ll need to fix this on our side

Zanie · Answer

Bumping the timeout may help in the meantime

Tim Enders · Answer

Awesome thank you

Zach Angell · Answer

Hey Tim thanks for the write up When did this error occur I m having trouble tracking down any server side error logs corresponding to your stack trace Could you also share a bit of detail about what your flow looks like Because the Prefect client is async sometimes we hit this error because the event loop is blocked rather than the server returning an error

Tim Enders · Answer

The timestamp on the errors is in Central time that is when they occurred I will shar some of my flow

Tim Enders · Answer

```dest dataset = os environ get GCP ACCOUNTS DATASET query = f select max CAST id AS INT64 as last id from ` dest dataset subscriptions orion` where api source = accounts highwater = query highwater query params = get params highwater full active load full load pages list = get pages list client subscriptions params try items = get items list map unmapped client unmapped subscriptions pages list except Exception as e return Failed Failed on an Exception sub data = for item in items sub data extend item result if not sub data return Completed message= No data from the API else transformed data = transform data sub data bqclient = bigquery Client bq result = load df bq bqclient transformed data bq result = None if isinstance bq result LoadJob and bq result state == DONE return Completed message= Load Finished elif isinstance bq result Failed slack webhook block = SlackWebhook load data pipeline notifications slack webhook block notify Hello from Prefect 2 0 return bq result else slack webhook block = SlackWebhook load data pipeline notifications slack webhook block notify Hello from Prefect 2 0 return Failed message= Load Failure breakpoint if name == main flow result = main ``` The error us ocurring during the mapping operation call to `get items list`

Zanie · Answer

How frequently do you encounter it Curious if you can reproduce against a local server

Tim Enders · Answer

I have about 3000 mapped tasks so I can t use a local server or I hit other errors

Tim Enders · Answer

Also should I be worried about seeing this occasionally ``` Orion logging error Traceback most recent call last File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages prefect logging handlers py line 146 in send logs await client create logs self pending logs File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages prefect client orion py line 1683 in create logs await f logs json=serialized logs File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpx client py line 1842 in post return await self request File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages httpx client py line 1527 in request return await self send request auth=auth follow redirects=follow redirects File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages prefect client base py line 182 in send response raise for status File home tenders cache pypoetry virtualenvs prefect orion HonJDUqB py3 10 lib python3 10 site packages prefect client base py line 125 in raise for status raise PrefectHTTPStatusError from httpx error exc from exc cause prefect exceptions PrefectHTTPStatusError Server error 500 Internal Server Error for url Response exception message Internal Server Error For more information check Worker information Approximate queue length 62 Pending log batch length 2866 Pending log batch size 1031760 The log worker will attempt to send these logs again in 2 0s```

Zanie · Answer

Generally no since we ll retry Good to have a report of it though as the server needs to be able to handle high log volumes

Tim Enders · Answer

Hmmm I set the timeout to 45 second and I have been getting crashes closed out my terminal and venv trying again but I am seeing this spam in the logs ```15 53 17 222 | ERROR | Task run Get Items d8ed86f1 56 Crash detected Execution was cancelled by the runtime environment 15 53 17 223 | ERROR | Task run Get Items d8ed86f1 2718 Crash detected Execution was cancelled by the runtime environment 15 53 17 224 | ERROR | Task run Get Items d8ed86f1 2738 Crash detected Execution was cancelled by the runtime environment 15 53 17 225 | ERROR | Task run Get Items d8ed86f1 61 Crash detected Execution was cancelled by the runtime environment 15 53 17 226 | ERROR | Task run Get Items d8ed86f1 47 Crash detected Execution was cancelled by the runtime environment 15 53 17 227 | ERROR | Task run Get Items d8ed86f1 46 Crash detected Execution was cancelled by the runtime environment 15 53 17 227 | ERROR | Task run Get Items d8ed86f1 148 Crash detected Execution was cancelled by the runtime environment 15 53 17 228 | ERROR | Task run Get Items d8ed86f1 2732 Crash detected Execution was cancelled by the runtime environment 15 53 17 229 | ERROR | Task run Get Items d8ed86f1 126 Crash detected Execution was cancelled by the runtime environment 15 53 17 230 | ERROR | Task run Get Items d8ed86f1 66 Crash detected Execution was cancelled by the runtime environment 15 53 17 231 | ERROR | Task run Get Items d8ed86f1 2695 Crash detected Execution was cancelled by the runtime environment 15 53 17 232 | ERROR | Task run Get Items d8ed86f1 3060 Crash detected Execution was cancelled by the runtime environment 15 53 17 234 | ERROR | Task run Get Items d8ed86f1 3087 Crash detected Execution was cancelled by the runtime environment 15 53 17 235 | ERROR | Task run Get Items d8ed86f1 140 Crash detected Execution was cancelled by the runtime environment 15 53 17 236 | ERROR | Task run Get Items d8ed86f1 77 Crash detected Execution was cancelled by the runtime environment 15 53 17 237 | ERROR | Task run Get Items d8ed86f1 3070 Crash detected Execution was cancelled by the runtime environment 15 53 17 238 | ERROR | Task run Get Items d8ed86f1 2736 Crash detected Execution was cancelled by the runtime environment 15 53 17 239 | ERROR | Task run Get Items d8ed86f1 135 Crash detected Execution was cancelled by the runtime environment 15 53 17 240 | ERROR | Task run Get Items d8ed86f1 2756 Crash detected Execution was cancelled by the runtime environment 15 53 17 241 | ERROR | Task run Get Items d8ed86f1 133 Crash detected Execution was cancelled by the runtime environment 15 53 17 241 | ERROR | Task run Get Items d8ed86f1 101 Crash detected Execution was cancelled by the runtime environment 15 53 17 242 | ERROR | Task run Get Items d8ed86f1 174 Crash detected Execution was cancelled by the runtime environment 15 53 17 243 | ERROR | Task run Get Items d8ed86f1 94 Crash detected Execution was cancelled by the runtime environment 15 53 17 244 | ERROR | Task run Get Items d8ed86f1 113 Crash detected Execution was cancelled by the runtime environment 15 53 17 245 | ERROR | Task run Get Items d8ed86f1 2782 Crash detected Execution was cancelled by the runtime environment 15 53 17 246 | ERROR | Task run Get Items d8ed86f1 49 Crash detected Execution was cancelled by the runtime environment 15 53 17 246 | ERROR | Task run Get Items d8ed86f1 2735 Crash detected Execution was cancelled by the runtime environment 15 53 17 247 | ERROR | Task run Get Items d8ed86f1 41 Crash detected Execution was cancelled by the runtime environment 15 53 17 248 | ERROR | Task run Get Items d8ed86f1 2762 Crash detected Execution was cancelled by the runtime environment 15 53 17 249 | ERROR | Task run Get Items d8ed86f1 62 Crash detected Execution was cancelled by the runtime environment ```

Matt Denno · Answer

Hey < Tim Enders> just wondering if you determined the cause of this issue I m seeing the same thing in one of my flows

Zanie · Answer

If you turn on DEBUG level logs we ll get a full traceback That message means that a `asyncio exceptions CancelledError` was raised though

Zanie · Answer

Sounds like we re just seeing that cancellation from your other traceback propagate through all the tasks

Zanie · Answer

Resolving these issues is likely my next top priority

Tim Enders · Answer

+1

Tim Enders · Answer

Let me know if you need anything more from me < Zanie> I had an emergency come up and have been offline for a couple of days