Hello I m on 0 1 4 19 local Server and UI Also have static D

Question

Hello I m on 0 1 4 19 local Server and UI Also have static Dask cluster installed with helm with microK8S Am able to successfully execute executor = DaskExecutor address= lt <tcp microk8s kub svr1 8786 7Ctcp lt my microk8s dask svr ip gt 8786 gt 22> flow run executor=executor on this remote Dask from python command line What do I need to do to 1 register the same flow with local Server i can do this 2 setup an agent which one and 3 execute this flow from within local Server UI to run on static Dask cluster Thanks

Kevin Kho · Answer

Hi < Mdu Keswa> you need to `prefect backend server` like and then use `flow register` instead of `flow run` You can also remove `flow run` and use the CLI like `prefect register `

Kevin Kho · Answer

Check the <https docs prefect io orchestration agents overview html agent types|agent types> for the one you need

Kevin Kho · Answer

and then you can either set a schedule or run immediately and it will use the DaskCluster attached to the executor in your Flow

Mdu Keswa · Answer

Thanks Kevin Is there an agent you can point me to still confused which one if I m using static Dask cluster

Nitin Karolla · Answer

Hi < Kevin Kho> I am working on something similar except it compulsory to have dask cluster Can I make it run on K8s jobs I have registered a flow but was not really able to run it on the UI

Kevin Kho · Answer

The local agent might work for you < Mdu Keswa>

Kevin Kho · Answer

< Nitin Karolla> are you using Prefect to run K8s jobs

Nitin Karolla · Answer

I mean I thought prefect would automatically launch k8s jobs as an executor does it not

Dylan · Answer

Hi < Nitin Karolla> check out

Dylan · Answer

We have 3 primary concepts in our system for configuring a flow run 1 Storage 2 Run Configs 3 Executors

Dylan · Answer

The Run Config is what determines how a flow run is executed

Dylan · Answer

Executors determine how task runs are executed

Dylan · Answer

In your case you d want a KubernetesRun config to create your k8s job and either a LocalExecutor or a LocalDaskExecutor to execute task runs inside your job

Dylan · Answer

LocalDaskExecutor comes with some out of the box parallelism

Dylan · Answer

Hi < Mdu Keswa> check out

Dylan · Answer

You can use any agent run config and connect to a static dask cluster for execution

Dylan · Answer

What infrastructure do you have access to

Nitin Karolla · Answer

Okay thanks for that < Dylan> I am curious agent is the one that orchestrates everything Would kubernetesRun still need a DaskExecutor

Dylan · Answer

Agents are only responsible for creating the infrastructure defined by the run config

Dylan · Answer

They poll for flow runs and then create infrastructure when runs are ready to be scheduled

Dylan · Answer

The executor defines how tasks execute within that infrastructure or using adjacent other infrastructure

Dylan · Answer

The flow run itself reports it state amp the state of its tasks to the backend Prefect Cloud or Prefect Server

Dylan · Answer

That doc is pretty comprehensive I think it will give you a good overview

Nitin Karolla · Answer

Thanks Dylan this so far has been helpful

Mdu Keswa · Answer

Tried with LocalAgent and Server seems to be connecting with DaskExecutor This sample job takes seconds when submitted to static Dask cluster directly But when ran from within UI it doesn t seem to finish See attached log screenshot

Kevin Kho · Answer

Do your tasks run successfully

Mdu Keswa · Answer

Hi Kevin it won t finish Interesting thing is bokeh view on Dask cluster for both types of execution See attached

Mdu Keswa · Answer

Here s another bokeh perspective

Kevin Kho · Answer

Maybe you can try <https github com PrefectHQ prefect issues 4353|this>