https://prefect.io logo
Title
e

Emma Rizzi

10/12/2022, 3:17 PM
Hello! I'm stumbing accross a weird error and have trouble debugging (error in thread) I'm migrating from gitlab.com registry to dockerhub for my flows DockerStorage, and a flow that used to work is not failing on dockerhub. It is running on a kubernetes cluster, and I tried to watch logs directly from the pod, the error does not even appear it justs stops the pod when starting the first non-parameter task. How can I get any more details on the error occuring ? I'm using prefect 1.2.4
1
Error : occuring when starting the first task (parameter tasks ran well)
Pod prefect-job-c4945354-gkspm failed. Container 'flow' state: terminated Exit Code:: 139 Reason: Error
k

Kalise Richmond

10/12/2022, 4:17 PM
Hi Emma, generally this means your OOM and the python process has crashed.
e

Emma Rizzi

10/13/2022, 8:57 AM
hi @Kalise Richmond, this errors happens right at the beginning of the flow, the first line starts with a log that does not show up, nothing has happened yet so it seems unlikely to be a memory issue... I noticed this in the logs right before the failure, and can't find anything in the doc about what it means :
Flow run RUNNING: terminal tasks are incomplete.
Any insights ?
i have reran the flow by watching resources and it never goes above 50% CPU and 10% memory. All flows that ran before migration have stopped working now, although i created a hello world flow with nothing but a log line that worked, I have no idea where to look
k

Kalise Richmond

10/13/2022, 5:53 PM
Hi Emma, gitlab is a script based storage whereas docker storage is not. How are you defining the docker storage? Do you have anything dynamically defined inside your flow (parameters, idempotency keys)?
e

Emma Rizzi

10/14/2022, 7:53 AM
Sorry if that was unclear, I'm using docker storage in both case and my flow's definition has not changed, I just changed the registry from gitlab.registry.com to docker.io and rebuilt the image. It seems to work again after changing my flows version from python 3.9 to 3.10, as I noticed that only my python 3.10 flows did not break during migration. That's a very weird though!