I have a flow run that I think has failed ish hop in the thr

Question

I have a flow run that I think has failed ish hop in the thread to check out the Prefect Cloud log message can you help me understand what the message it telling me

Cooper Marcus · Answer

``` 04 24 37 WARNING CloudFlowRunner Error getting flow run info Traceback most recent call last File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages urllib3 connectionpool py line 699 in urlopen httplib response = self make request File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages urllib3 connectionpool py line 382 in make request self validate conn conn File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages urllib3 connectionpool py line 1010 in validate conn conn connect File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages urllib3 connection py line 416 in connect self sock = ssl wrap socket File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages urllib3 util ssl py line 449 in ssl wrap socket ssl sock = ssl wrap socket impl File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages urllib3 util ssl py line 493 in ssl wrap socket impl return ssl context wrap socket sock server hostname=server hostname File home ec2 user SageMaker pyenv versions 3 8 6 lib python3 8 ssl py line 500 in wrap socket return self sslsocket class create File home ec2 user SageMaker pyenv versions 3 8 6 lib python3 8 ssl py line 1040 in create self do handshake File home ec2 user SageMaker pyenv versions 3 8 6 lib python3 8 ssl py line 1309 in do handshake self sslobj do handshake ssl SSLEOFError EOF occurred in violation of protocol ssl c 1124 During handling of the above exception another exception occurred Traceback most recent call last File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages requests adapters py line 439 in send resp = conn urlopen File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages urllib3 connectionpool py line 783 in urlopen return self urlopen File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages urllib3 connectionpool py line 783 in urlopen return self urlopen File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages urllib3 connectionpool py line 783 in urlopen return self urlopen Previous line repeated 3 more times File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages urllib3 connectionpool py line 755 in urlopen retries = retries increment File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages urllib3 util retry py line 574 in increment raise MaxRetryError pool url error or ResponseError cause urllib3 exceptions MaxRetryError HTTPSConnectionPool host= port=443 Max retries exceeded with url graphql Caused by SSLError SSLEOFError 8 EOF occurred in violation of protocol ssl c 1124 During handling of the above exception another exception occurred Traceback most recent call last File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages prefect engine cloud flow runner py line 188 in interrupt if cancelling flow run info = self client get flow run info flow run id File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages prefect client client py line 1145 in get flow run info result = self graphql query data flow run by pk type ignore File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages prefect client client py line 298 in graphql result = File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages prefect client client py line 213 in post response = self request File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages prefect client client py line 459 in request response = self send request File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages prefect client client py line 351 in send request response = File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages requests sessions py line 590 in post return self request POST url data=data json=json kwargs File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages requests sessions py line 542 in request resp = self send prep send kwargs File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages requests sessions py line 655 in send r = adapter send request kwargs File home ec2 user SageMaker PGE Dx Risk venv lib python3 8 site packages requests adapters py line 514 in send raise SSLError e request=request requests exceptions SSLError HTTPSConnectionPool host= port=443 Max retries exceeded with url graphql Caused by SSLError SSLEOFError 8 EOF occurred in violation of protocol ssl c 1124 ```

Kevin Kho · Answer

Hey I believe this is related to a slight outage that we had on Oct 7 5 20 something ET that lasted for maybe 30 seconds The certificates have been changed and this is not expected going forward

Marko Jamedzija · Answer

What s the reason this could happen in general I m getting the similar error that doesn t make my flow finish but just stuck running forever Here s the log ``` 2021 10 10 19 10 51+0000 WARNING prefect CloudFlowRunner | Error getting flow run info Traceback most recent call last File usr local lib python3 7 site packages urllib3 connectionpool py line 445 in make request six raise from e None File lt string gt line 3 in raise from File usr local lib python3 7 site packages urllib3 connectionpool py line 440 in make request httplib response = conn getresponse File usr local lib python3 7 http client py line 1373 in getresponse response begin File usr local lib python3 7 http client py line 319 in begin version status reason = self read status File usr local lib python3 7 http client py line 280 in read status line = str self fp readline MAXLINE + 1 iso 8859 1 File usr local lib python3 7 socket py line 589 in readinto return self sock recv into b socket timeout timed out During handling of the above exception another exception occurred Traceback most recent call last File usr local lib python3 7 site packages requests adapters py line 449 in send timeout=timeout File usr local lib python3 7 site packages urllib3 connectionpool py line 756 in urlopen method url error=e pool=self stacktrace=sys exc info 2 File usr local lib python3 7 site packages urllib3 util retry py line 532 in increment raise six reraise type error error stacktrace File usr local lib python3 7 site packages urllib3 packages six py line 770 in reraise raise value File usr local lib python3 7 site packages urllib3 connectionpool py line 706 in urlopen chunked=chunked File usr local lib python3 7 site packages urllib3 connectionpool py line 447 in make request self raise timeout err=e url=url timeout value=read timeout File usr local lib python3 7 site packages urllib3 connectionpool py line 337 in raise timeout self url Read timed out read timeout= s timeout value urllib3 exceptions ReadTimeoutError HTTPConnectionPool host= prefect apollo prefect port=4200 Read timed out read timeout=15 During handling of the above exception another exception occurred Traceback most recent call last File usr local lib python3 7 site packages prefect engine cloud flow runner py line 188 in interrupt if cancelling flow run info = self client get flow run info flow run id File usr local lib python3 7 site packages prefect client client py line 1562 in get flow run info result = self graphql query data flow run by pk type ignore File usr local lib python3 7 site packages prefect client client py line 554 in graphql retry on api error=retry on api error File usr local lib python3 7 site packages prefect client client py line 458 in post retry on api error=retry on api error File usr local lib python3 7 site packages prefect client client py line 738 in request session=session method=method url=url params=params headers=headers File usr local lib python3 7 site packages prefect client client py line 606 in send request timeout=prefect context config cloud request timeout File usr local lib python3 7 site packages requests sessions py line 590 in post return self request POST url data=data json=json kwargs File usr local lib python3 7 site packages requests sessions py line 542 in request resp = self send prep send kwargs File usr local lib python3 7 site packages requests sessions py line 655 in send r = adapter send request kwargs File usr local lib python3 7 site packages requests adapters py line 529 in send raise ReadTimeout e request=request ``` I have started having situations like these quite often The flow executes only 3 `RunNamespacedJob` tasks in parallel The first two finish and the last one doesn t get a confirmation that is done from time to time and gets stuck even though the underlying k8s job finished Maybe I need to increase this read timeout

Kevin Kho · Answer

This is server right We can try increasing the read timeout like you suggested Have you seen how I can look if you re not familiar

Marko Jamedzija · Answer

Yep this is on server slightly smiling face I also didn t check how to do it yet slightly smiling face So if you already know where to look it would be great slightly smiling face

Kevin Kho · Answer

Yeah one sec

Marko Jamedzija · Answer

pray

Kevin Kho · Answer

Check <https github com PrefectHQ prefect pull 4119 files|this> You need to set `prefect context config cloud request timeout`

Marko Jamedzija · Answer

Thanks So I guess I can set this as an env to the k8s job definition for the k8s run config For this it would be then `PREFECT CONTEXT CONFIG CLOUD REQUEST TIMEOUT` Sorry I haven t set any of these so far slightly smiling face I m not sure if I should put `CONFIG` <https docs prefect io core concepts configuration html environment variables|this doc> is a bit ambiguous slightly smiling face

Kevin Kho · Answer

that looks right to me

Marko Jamedzija · Answer

I ll try it tomorrow and let you know if it worked slightly smiling face Thanks again

Marko Jamedzija · Answer

I looked how the env vars are set in official prefect helm chart and deducted that this one should be `PREFECT CLOUD REQUEST TIMEOUT` slightly smiling face I tested it and the value is set correctly I ll see if it helps to mitigate the problem slightly smiling face

Kevin Kho · Answer

Oh ok

Marko Jamedzija · Answer

In the end upon inspecting the code I concluded that this timeout is not the cause of the problem The logs above are just warnings and don t cause the flow to be stuck In this particular scenario the flow was stuck in running state because the `RunNamespacedJob` was stuck in running state even though the underlying pod finished long time ago And these warnings started coming like 5 hours after that so it s not the cause of an issue

Marko Jamedzija · Answer

Any other ides why this task might get stuck in the running state

Kevin Kho · Answer

2 ideas I saw one person had an issue with unclosed connections second does it fail on a task mapped over Dask

Marko Jamedzija · Answer

We use `LocalDaskExecutor processes ` for executing these 3 `RunNamespacedJobs` tasks in parallel slightly smiling face Do you know how I could check this unclosed connections Thanks slightly smiling face

Kevin Kho · Answer

I do not But there was a guy in community who said he had containers not shutting down and then he found out it was a snowflake connector that didn t close in the code base So I guess you need to check like that

Marko Jamedzija · Answer

Hmmm slightly smiling face I ll check it out We do access Snowflake in these containers But on the other hand the k8s jobs finish and pods get deleted meaning the containers shut down just the the Prefect flow k8s runner job doesn t and it still sees one task as a running one