https://prefect.io
Join Slack
Hello, I'm having a very frustrating problem with running prefect on our Google Cloud Platform Kuber...
# ask-community
s
Hello, I'm having a very frustrating problem with running prefect on our Google Cloud Platform Kubernetes Cluster and and hoping you can help me out. In short, a flow that was running fine on Thursday of last week suddenly gave vague errors on Friday when I tried to push an update. Is there a way I can get more details on the error from Kubernetes so I can debug this more efficiently? (logs and more examples will be posted in the replies shortly).
For context, we have a Kubernetes cluster running constantly that serves as our main Prefect agent. To push code there, we create a Docker container, push it to Google-Container-Registry, then register the flow in Prefect as a 
KubernetesRun
with the associated image url as the 
image
argument. I can provide code snippets here if necessary. However, ever since Friday I get a very vague error in the prefect logs right after triggering the flow:
Copy code
Pod prefect-job-511614c7-56n5v failed.
	Container 'flow' state: terminated
		Exit Code:: 2
		Reason: Error
How can I get more insight into what is going on?
n
Hi @Scott White! Hmm, have you checked the kubernetes logs like 
kubectl get events
for any sort of kubernetes specific weirdness?
c
Hi Scott, can you check the logs of the pod that died from Kubernetes? If the pod / container died, we’d need to see that:
Copy code
kubectl logs prefect-job-511614c7-56n5v --previous
s
Thanks I will check
I don't see anything wrong with kubernetecs, and older flows that I contariner-ized and registered weeks ago still work just fine with this cluster as their agent. is there is any potential version conflict with the prefect 2.0 upgrade?
n
the 2.0 upgrade would certainly be a breaking change if you happened to be pulling the latest version of prefect anywhere in your registration process however if your flows (and environment where your agent lives) are still on 
prefect>=1.0.0, <2.0.0
then there shouldn't be any impact from the release of 2.0
s
Ok I will verify that they are <2.0, I am suspecting that is the issue
👍 1
w
@Scott White Hi scott, sorry to drop in on your thread, but I am facing a similar issue. Did you figure out what was happening here?
s
Yes I needed to pin prefect<2.0 and everything worked fine
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by