https://prefect.io logo
Title
j

José Duarte

08/22/2022, 2:28 PM
I have the following flow:
@flow
def test():
    print("hello world")
And it will fail retro-actively from time to time:
15:18:12.953 | INFO    | prefect.agent - Submitting flow run '8b6a74ba-6dbe-49bd-b303-d5e9c863a087'
15:18:13.097 | INFO    | prefect.infrastructure.process - Opening process 'dangerous-ostrich'...
15:18:13.104 | INFO    | prefect.agent - Completed submission of flow run '8b6a74ba-6dbe-49bd-b303-d5e9c863a087'
15:18:30.922 | ERROR   | Flow run 'dangerous-ostrich' - Crash detected! Execution was interrupted by an unexpected exception.
15:18:30.954 | INFO    | prefect.engine - Engine execution of flow run '8b6a74ba-6dbe-49bd-b303-d5e9c863a087' aborted by orchestrator: This run has already terminated.
hello world
15:18:35.274 | INFO    | prefect.infrastructure.process - Process 'dangerous-ostrich' exited cleanly.
As you can see above, it completed successfully but at the same time it didn’t. Furthermore, the code clearly ran (
hello world
was printed), but it was still marked as an error. Shouldn’t the flow just be marked as completed?
1
k

Khuyen Tran

08/22/2022, 2:51 PM
which version of Prefect are you in?
j

José Duarte

08/22/2022, 2:53 PM
@Khuyen Tran
Version:             2.1.0
API version:         0.8.0
Python version:      3.8.9
Git commit:          30ca715c
Built:               Wed, Aug 17, 2022 5:13 PM
OS/Arch:             darwin/x86_64
Profile:             default
Server type:         hosted
k

Khuyen Tran

08/22/2022, 2:53 PM
I recommend upgrading to 2.1.1, which fixes some issues in 2.1.0
and try running again
j

José Duarte

08/22/2022, 2:54 PM
Alright, will do!
r

Rio McMahon

09/14/2022, 3:31 PM
Did this get resolved? I am running into the same error with prefect 2.3.2
j

José Duarte

09/14/2022, 3:32 PM
@Rio McMahon After the update, I haven’t ran into the issue again
🙌 2
n

Naila Chennit

09/23/2022, 1:41 PM
Hello @Khuyen Tran, i'mrunning my flows using 2.3.1 version and still having the same issue from time to time it crashs for no apparent reason even when the flow did his job and completed it
j

Jeff Hale

09/23/2022, 1:52 PM
Hi Naila. If you update to 2.4.1 does the issue go away?
n

Naila Chennit

09/23/2022, 1:55 PM
Hi Jeff , didn't test it , but i'll try , thank you
r

Rio McMahon

09/23/2022, 3:40 PM
@Naila Chennit In my case it ended up being operator error on my part; I had an agent running locally and in the cloud - both were polling for the same work queue tag. Removing the local agent fixed it.
👍 3
:thank-you: 3
n

Naila Chennit

09/26/2022, 12:14 PM
Thank you @Rio McMahon in my case, i solved the issue by seting the number of pod replicas to 1 instead of 2 as i'm using k8 job as infrastructure but i gess it was the same problem as yours as the two pods were polling from the same work queue
👍 1
🎉 1