https://prefect.io logo
Join Slack
Powered by
Hello, I'm on 0.1.4.19 local Server and UI. Also h...
# prefect-server
m
Hello, I'm on 0.1.4.19 local Server and UI. Also have, static Dask cluster installed with helm with microK8S. Am able to successfully execute "... executor = DaskExecutor(address="<tcp://microk8s-kub-svr1:8786%7Ctcp://&lt;my-microk8s-dask-svr-ip&gt;:8786&gt;%22) flow.run(executor=executor)" on this remote Dask from python command-line. What do I need to do to 1) register the same flow with local Server (i can do this) 2) setup an agent (which one?), and 3) execute this flow from within local Server UI to run on static Dask cluster? Thanks.
k
Hi @Mdu Keswa, you need to 
prefect backend server
like this and then use 
flow.register
instead of 
flow.run
. You can also remove 
flow.run
and use the CLI like 
prefect register …
Check the agent types for the one you need?
and then you can either set a schedule or run immediately and it will use the DaskCluster attached to the executor in your Flow
m
Thanks Kevin! Is there an agent you can point me to - still confused which one if I'm using static Dask cluster.
n
Hi @Kevin Kho, I am working on something similar, except it compulsory to have dask cluster? Can I make it run on K8s jobs? I have registered a flow but was not really able to run it on the UI.
k
The local agent might work for you @Mdu Keswa!
@Nitin Karolla, are you using Prefect to run K8s jobs?
n
I mean, I thought prefect would automatically launch k8s jobs as an executor does it not?
d
We have 3 primary concepts in our system for configuring a flow run: 1. Storage 2. Run Configs 3. Executors
The Run Config is what determines how a flow run is executed
Executors determine how task runs are executed
In your case, you’d want a KubernetesRun config to create your k8s job and either a LocalExecutor or a LocalDaskExecutor to execute task runs inside your job
LocalDaskExecutor comes with some out-of-the-box parallelism
You can use any agent/run config and connect to a static dask cluster for execution
What infrastructure do you have access to?
n
Okay, thanks for that @Dylan. I am curious, agent is the one that orchestrates everything, Would kubernetesRun still need a DaskExecutor?
d
Agents are only responsible for creating the infrastructure defined by the run config
They poll for flow runs and then create infrastructure when runs are ready to be scheduled
The executor defines how tasks execute within that infrastructure (or using adjacent/other infrastructure)
The flow run itself reports it state & the state of its tasks to the backend (Prefect Cloud or Prefect Server)
That doc is pretty comprehensive, I think it will give you a good overview
n
Thanks Dylan, this so far has been helpful
🙌 1
m
Tried with LocalAgent and Server seems to be connecting with DaskExecutor. This sample job takes seconds when submitted to static Dask cluster directly. But when ran from within UI, it doesn't seem to finish. See attached log screenshot.
k
Do your tasks run successfully?
m
Hi @Kevin, it won't finish. Interesting thing is 'bokeh; view on Dask cluster for both types of execution. See attached.
Here's another 'bokeh' perspective:
k
Maybe you can try this
2 Views
Open in Slack
PreviousNext
Powered by