Hey all Does anyone know off hand why a flow wouldn t get pi

Question

Hey all Does anyone know off hand why a flow wouldn t get picked up after it s been submitted The last version of it got picked up I ve deleted the flow bumped the version tag rebuilt and still nothing Get lazarus killed 3 times and errors out Agent is running other flows fine

Brian Mesick · Answer

This is via Cloud fwiw

james.lamb · Answer

9 out of 10 times that I ve had this problem it has been because of a mismatch in labels Check the labels on your agent and on the flow

james.lamb · Answer

One way I ve been caught with this is changing the type of `Storage` between versions of a flow The labels for the flow will change based on the `Storage` you use

Brian Mesick · Answer

Ah it s probably the latter I guess I added retries to a task then had to add a result store for an upstream task which I guess would change the storage

Brian Mesick · Answer

Tags are the same as we ve always used

Brian Mesick · Answer

Hm I used a `PrefectResult` to store this no new labels seem to have been added The only label is the one we ve always run with `prod`

Chris White · Answer

Usually this is a symptom of something going wrong in your execution environment for example an image pull error or an unschedulable pod we re working on enhancements to the various agents to elevate these errors

Brian Mesick · Answer

Interesting ok Looking now I see that it registering the flow only pushed ~1mb to ECR which is usually the first step then the second is to push the whole 191mb

Brian Mesick · Answer

< Dylan> ^ thanks

Dylan · Answer

< Brian Mesick> are you running on Kubernetes

Brian Mesick · Answer

Yes

Dylan · Answer

Can you find container logs for your submitted flow runs

Dylan · Answer

In GKE when I run into a problem like this usually I can find the dead job

Dylan · Answer

and the audit logs give me some good clues as to what s going on

Brian Mesick · Answer

It never seemed to spin up containers no pods or jobs showed up

Brian Mesick · Answer

Or if they spun up they were cleaned up before I could find them

Dylan · Answer

If you have a resource manager running from your Prefect k8s agent try disabling it

Dylan · Answer

the dead jobs should show up

Dylan · Answer

then you can look at their logs

Dylan · Answer

otherwise try digging into the prefect agent s deployment logs

Brian Mesick · Answer

In the agent logs we just see the `Found 1 flow run s ` then `Deploying flow run ` where the guid matches the run that I would expect Let me try to figure out how to disable the resource manager here Any idea why those logs wouldn t percolate up

Dylan · Answer

The logs from the flow run itself should get sent to cloud

Dylan · Answer

Since they re run on a different node than the agent they wouldn t appear in the agent container

Dylan · Answer

I was looking to see if the agent was throwing errors but it appears everything is working properly

Brian Mesick · Answer

Right but they I would expect them to show up in Cloud

Brian Mesick · Answer

But I guess if it s not bootstrapping correctly it couldn t get there

Dylan · Answer

Correct

Dylan · Answer

this sounds like the job is dying

Dylan · Answer

Which for me is usually an imagePullBackoff error

Dylan · Answer

But that s hard to see right now

Dylan · Answer

see Chris s comment about improving this logging experience

Dylan · Answer

We re going to do some more work on agents so they re a little more involved

Dylan · Answer

Hopefully that will help with this issue

Brian Mesick · Answer

Cool better logging around this would definitely be helpful It sounds like we ve run into it before

Dylan · Answer

Agreed Let me know if you get access to the job logs

Brian Mesick · Answer

Going to be a bit we need to push through a pr to turn off the resource mgr on the cluster

Brian Mesick · Answer

So I m seeing what looks like the same error as someone else had on here a couple of months ago `AttributeError SSLSocket object has no attribute connection ` which they fixed by pinning the Snowflake adapter version

Brian Mesick · Answer

I m not sure why that would suddenly show up for us now in the middle of several dev cycles on the same flow without changing versions of anything

Brian Mesick · Answer

But we are on old versions of Prefect in the agent and containers

Dylan · Answer

Are you using different versions of Prefect for development and in the containers

Brian Mesick · Answer

Maybe but this flow was running in this container `prefecthq prefect 0 12 5 python3 8` a day or two ago

Brian Mesick · Answer

Minus the few changes I ve been focused on swiching from backoff to prefect retry signal being the big one I m guessing some other dependency may have updated and bumped the urllib3 requirement

Dylan · Answer

I m glad you were able to diagnose the issue I believe our containers still `pip install` dependencies If so something may have changed as you pushed up a new version of your container while working on the flow

Brian Mesick · Answer

< Dylan> I was able to get past my issue by pinning urllib to Prefect s lowest pinned version but I think it would probably make sense for you all to pin the max version that will actually work as well there is currently no upper bound as that would have made this whole thing pretty obvious

Dylan · Answer

Thanks for that suggestion