Hello I m having a very frustrating problem with running pre Prefect Community #prefect-community

Hello, I'm having a very frustrating problem with ...

Scott White

08/01/2022, 12:13 PM

Hello, I'm having a very frustrating problem with running prefect on our Google Cloud Platform Kubernetes Cluster and and hoping you can help me out. In short, a flow that was running fine on Thursday of last week suddenly gave vague errors on Friday when I tried to push an update. Is there a way I can get more details on the error from Kubernetes so I can debug this more efficiently? (logs and more examples will be posted in the replies shortly).

Scott White

08/01/2022, 12:16 PM

For context, we have a Kubernetes cluster running constantly that serves as our main Prefect agent. To push code there, we create a Docker container, push it to Google-Container-Registry, then register the flow in Prefect as a

KubernetesRun

with the associated image url as the

image

argument. I can provide code snippets here if necessary. However, ever since Friday I get a very vague error in the prefect logs right after triggering the flow:

Copy code

Pod prefect-job-511614c7-56n5v failed.
	Container 'flow' state: terminated
		Exit Code:: 2
		Reason: Error

How can I get more insight into what is going on?

Nate

08/01/2022, 2:56 PM

Hi @Scott White! Hmm, have you checked the kubernetes logs like

kubectl get events

for any sort of kubernetes specific weirdness?

Christopher Boyd

08/01/2022, 2:56 PM

Hi Scott, can you check the logs of the pod that died from Kubernetes? If the pod / container died, we’d need to see that:

Copy code

kubectl logs prefect-job-511614c7-56n5v --previous

Scott White

08/01/2022, 3:00 PM

Thanks I will check

Scott White

08/01/2022, 6:47 PM

I don't see anything wrong with kubernetecs, and older flows that I contariner-ized and registered weeks ago still work just fine with this cluster as their agent. is there is any potential version conflict with the prefect 2.0 upgrade?

Nate

08/01/2022, 6:55 PM

the 2.0 upgrade would certainly be a breaking change if you happened to be pulling the latest version of prefect anywhere in your registration process however if your flows (and environment where your agent lives) are still on

prefect>=1.0.0, <2.0.0

then there shouldn't be any impact from the release of 2.0

Scott White

08/01/2022, 6:56 PM

Ok I will verify that they are <2.0, I am suspecting that is the issue

👍 1

Walter Cavinaw

08/30/2022, 1:08 AM

@Scott White Hi scott, sorry to drop in on your thread, but I am facing a similar issue. Did you figure out what was happening here?

Scott White

08/30/2022, 12:08 PM

Yes I needed to pin prefect<2.0 and everything worked fine

2 Views

Open in Slack

Previous Next