Channels
#prefect-community
Title
e
Emma Rizzi
10/12/2022, 3:17 PMHello! I'm stumbing accross a weird error and have trouble debugging (error in thread)
I'm migrating from gitlab.com registry to dockerhub for my flows DockerStorage, and a flow that used to work is not failing on dockerhub. It is running on a kubernetes cluster, and I tried to watch logs directly from the pod, the error does not even appear it justs stops the pod when starting the first non-parameter task. How can I get any more details on the error occuring ?
I'm using prefect 1.2.4
✅ 1
Error : occuring when starting the first task (parameter tasks ran well)
Copy code
Pod prefect-job-c4945354-gkspm failed. Container 'flow' state: terminated Exit Code:: 139 Reason: Errork
Kalise Richmond
10/12/2022, 4:17 PMHi Emma, generally this means your OOM and the python process has crashed.
e
Emma Rizzi
10/13/2022, 8:57 AMhi @Kalise Richmond, this errors happens right at the beginning of the flow, the first line starts with a log that does not show up, nothing has happened yet so it seems unlikely to be a memory issue... I noticed this in the logs right before the failure, and can't find anything in the doc about what it means :
Flow run RUNNING: terminal tasks are incomplete.i have reran the flow by watching resources and it never goes above 50% CPU and 10% memory. All flows that ran before migration have stopped working now, although i created a hello world flow with nothing but a log line that worked, I have no idea where to look
k
Kalise Richmond
10/13/2022, 5:53 PMHi Emma, gitlab is a script based storage whereas docker storage is not. How are you defining the docker storage? Do you have anything dynamically defined inside your flow (parameters, idempotency keys)?
e
Emma Rizzi
10/14/2022, 7:53 AMSorry if that was unclear, I'm using docker storage in both case and my flow's definition has not changed, I just changed the registry from gitlab.registry.com to docker.io and rebuilt the image.
It seems to work again after changing my flows version from python 3.9 to 3.10, as I noticed that only my python 3.10 flows did not break during migration. That's a very weird though!