https://prefect.io logo
#prefect-community
Title
# prefect-community
s

Scott White

08/01/2022, 12:13 PM
Hello, I'm having a very frustrating problem with running prefect on our Google Cloud Platform Kubernetes Cluster and and hoping you can help me out. In short, a flow that was running fine on Thursday of last week suddenly gave vague errors on Friday when I tried to push an update. Is there a way I can get more details on the error from Kubernetes so I can debug this more efficiently? (logs and more examples will be posted in the replies shortly).
For context, we have a Kubernetes cluster running constantly that serves as our main Prefect agent. To push code there, we create a Docker container, push it to Google-Container-Registry, then register the flow in Prefect as a
KubernetesRun
with the associated image url as the
image
argument. I can provide code snippets here if necessary. However, ever since Friday I get a very vague error in the prefect logs right after triggering the flow:
Copy code
Pod prefect-job-511614c7-56n5v failed.
	Container 'flow' state: terminated
		Exit Code:: 2
		Reason: Error
How can I get more insight into what is going on?
n

Nate

08/01/2022, 2:56 PM
Hi @Scott White! Hmm, have you checked the kubernetes logs like
kubectl get events
for any sort of kubernetes specific weirdness?
c

Christopher Boyd

08/01/2022, 2:56 PM
Hi Scott, can you check the logs of the pod that died from Kubernetes? If the pod / container died, we’d need to see that:
Copy code
kubectl logs prefect-job-511614c7-56n5v --previous
s

Scott White

08/01/2022, 3:00 PM
Thanks I will check
I don't see anything wrong with kubernetecs, and older flows that I contariner-ized and registered weeks ago still work just fine with this cluster as their agent. is there is any potential version conflict with the prefect 2.0 upgrade?
n

Nate

08/01/2022, 6:55 PM
the 2.0 upgrade would certainly be a breaking change if you happened to be pulling the latest version of prefect anywhere in your registration process however if your flows (and environment where your agent lives) are still on
prefect>=1.0.0, <2.0.0
then there shouldn't be any impact from the release of 2.0
s

Scott White

08/01/2022, 6:56 PM
Ok I will verify that they are <2.0, I am suspecting that is the issue
👍 1
w

Walter Cavinaw

08/30/2022, 1:08 AM
@Scott White Hi scott, sorry to drop in on your thread, but I am facing a similar issue. Did you figure out what was happening here?
s

Scott White

08/30/2022, 12:08 PM
Yes I needed to pin prefect<2.0 and everything worked fine
2 Views