Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
e
Emma Rizzi
10/12/2022, 3:17 PMHello! I'm stumbing accross a weird error and have trouble debugging (error in thread)
I'm migrating from gitlab.com registry to dockerhub for my flows DockerStorage, and a flow that used to work is not failing on dockerhub. It is running on a kubernetes cluster, and I tried to watch logs directly from the pod, the error does not even appear it justs stops the pod when starting the first non-parameter task. How can I get any more details on the error occuring ?
I'm using prefect 1.2.4
✅ 1
Error : occuring when starting the first task (parameter tasks ran well)
Pod prefect-job-c4945354-gkspm failed. Container 'flow' state: terminated Exit Code:: 139 Reason: Errork
Kalise Richmond
10/12/2022, 4:17 PMHi Emma, generally this means your OOM and the python process has crashed.
e
Emma Rizzi
10/13/2022, 8:57 AMhi @Kalise Richmond, this errors happens right at the beginning of the flow, the first line starts with a log that does not show up, nothing has happened yet so it seems unlikely to be a memory issue... I noticed this in the logs right before the failure, and can't find anything in the doc about what it means :
Flow run RUNNING: terminal tasks are incomplete.i have reran the flow by watching resources and it never goes above 50% CPU and 10% memory. All flows that ran before migration have stopped working now, although i created a hello world flow with nothing but a log line that worked, I have no idea where to look
k
Kalise Richmond
10/13/2022, 5:53 PMHi Emma, gitlab is a script based storage whereas docker storage is not. How are you defining the docker storage? Do you have anything dynamically defined inside your flow (parameters, idempotency keys)?
e
Emma Rizzi
10/14/2022, 7:53 AMSorry if that was unclear, I'm using docker storage in both case and my flow's definition has not changed, I just changed the registry from gitlab.registry.com to docker.io and rebuilt the image.
It seems to work again after changing my flows version from python 3.9 to 3.10, as I noticed that only my python 3.10 flows did not break during migration. That's a very weird though!