Hello, I have a flow that I run on my laptop using DockerContainer infrastructure and takes around 4 minutes to complete. When I run that same flow on a kubernetes cluster using KubernetesJob it times out after 30 minutes (limit that I set in my flow). Any tips on how I could troubleshoot what is going on? I am using the default job template.

does it run, or does it stay in pending? Is it actually executing? What size nodes is your cluster using? With the default job template, there should be no default cpu / memory parameters When you run it locally, how many resources are being given to the docker container?

yeah, it runs until it times out

Finished in state TimedOut('Flow run exceeded timeout of 1800.0 seconds

. the nodes are 8 cores 32 Gb. The flow calls a DS model to get some predictions and it looks like that is what it is taking most of the time. the model is ~500 Mb

I’d look at the metrics of the cluster and see what kind of utilization / load it’s under when this flow executes. There are no “prefect performance” issues specifically, it’s just an execution environment for a job. A good recommendation though if you know the size / resource constraints, would be to use a customized job template and assign min/max cpu/memory constraints

yeah it is computing, I added additional logging to confirm

if you aren’t already using it, you could considering installing prometheus / grafana, so you can track performance / cluster health

would this be the right way of adding requests to my jobs without defining a custom manifest

customizations=[{
                "op": "add",
                "path": "/spec/template/spec/resources",
                "value": {"requests": {"memory": "2Gi", "cpu": "2"}},
            }]
            infrastructure = KubernetesJob(
                image = image,
                customizations = customizations,
                finished_job_ttl = 1*60*60, # one hour
            )

I have some notes and examples on this , but I need a few minutes to get them back to you

no rush, thanks for you help. you guys are awesome! great community!

I think you got it right, this is what I have in my notes for the customization:

customizations=[
    {
        "op": "add",
        "path": "/spec/imagePullSecrets",
        "value": [{'name': 'dockerhub'}],
    },
    {
        "op": "add",
        "path": "/spec/template/spec/resources",
        "value": {"limits": {"memory": "8Gi", "cpu": "4000m"}},
    }
    ],

thanks!