We've run into another issue that I was wondering if anyone could help clarify. I am running a dask cluster on an on-premise kubernetes cluster and I have the kubernetes agent running in the same namespace. The agent is connected to prefect cloud and the jobs are running on the dask cluster successfully. However, the agent never reports any runs. Is there something else that is needed in order to connect the agent with flows executed on a dask cluster?
k
Kevin Kho
03/28/2022, 2:14 PM
How are these flows runs? Were they created with a schedule?
b
Bennett Lambert
03/28/2022, 2:16 PM
I am executing the following:
Copy code
from prefect import task, Flow
from prefect.executors import DaskExecutor
from prefect.run_configs import KubernetesRun
@task
def get_value():
return "Example!"
@task
def output_value(value):
print(value)
with Flow("Test") as flow:
value = get_value()
output_value(value)
flow.run_config = KubernetesRun(labels=None)
flow.executor = DaskExecutor("IP_ADDRESS:8786")
flow.run()
k
Kevin Kho
03/28/2022, 2:19 PM
flow.run()
does not respect the
run_config
and is just meant for testing. For agent-backed runs, you need to register the Flow with Prefect Cloud/Server and then run it from the UI or schedule or by hitting the API of the backend. Those will get picked up and deployed by the agent
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.