Hey folks! Seeing a strange issue when using

KubernetesRun

amd

DaskExecutor (KubeCluster)

on AKS. I submit my flow and it runs, I can see the Dask Scheduler pod stand up, but then nothing happens. For example, the flow image provided shows a simple flow that has been running for 17(now 20)+ hours 😮 You can find the flow definition here https://github.com/pangeo-forge/pangeo-forge-azure-bakery/blob/add-k8s-cluster/flow_test/manual_flow.py And you can find the agent conf here https://github.com/pangeo-forge/pangeo-forge-azure-bakery/blob/add-k8s-cluster/prefect_agent_conf.yaml

Maybe the requested resources are not available? For example, when requesting 4GB of RAM, but the node on which the prefect agent runs only provides less than that per pod, than prefect might wait until resources are available, which is never the case 🤷

I thought that, but it actually spins off a new pod to be able to run the scheduler

can you try it again with:

def make_cluster():
    return KubeCluster(
        make_pod_spec(
            image=os.environ["AZURE_BAKERY_IMAGE"],
            labels={
                "flow": flow_name
            },
            memory_limit=None,
            memory_request=None
        )
    )

Can certainly give it a go

I have no experience with AKS, but removing all resource requests would exclude that root cause from the problem space.

Still similar behaviour, or at least, for now. Included the events for my workspace, you can see me cancel the long running one at the bottom, with the new dask worker & prefect job being spun up nearer the top

Hmm, I think prefect support would be appreciated here 🤷

Yeah haha, I think someone is on the case, but thanks for giving it a go!

Hello @ciaran! Do you see any logs for this flow?

Hey @Mariia Kerimova here's the dump of them 🙂

Do you use

dask

with e.g.

da.arange

or something similar by any chance?

Nope, just a

logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello, Cloud")
    return "hello result"

Just letting you know that I could reproduce your issue, but still investigating what's going on.

Well that's always a good sign if it can be reproduced 😅

Wow, it wasn't obvious to me. Thank you so much for sharing. I'm going to test it, and we should mention this in docs.

I opened an issue and tagged you. Thanks @ciaran and feel free to chime in there!

Same issue. I had workflows running until 29th April. Workflow as simple as this is not working.

import prefect

from prefect import task, Flow

from prefect.storage import GitLab

from prefect.run_configs import KubernetesRun

@task

def say_hello():

logger = prefect.context.get("logger")

<http://logger.info|logger.info>("Hello, Cloud!")

print('Hello, Cloud!')

with Flow("hello-world") as flow:

say_hello()

flow.storage = GitLab(

repo="repo/location",

path="path_to_workflow.py",

access_token_secret="secret"

)

flow.run_config = KubernetesRun(labels=['some-label'])