Javier Domingo Cansino
03/23/2021, 5:25 PMMariia Kerimova
03/23/2021, 6:49 PMprefect agent kubernetes start --name AgentName --token <runner token> --label <label-matching-flow-label>
.
Here is a simple flow you can use to test your setup:
from prefect import task, Flow
from prefect.storage import Docker
from prefect.run_configs import KubernetesRun
import random
from time import sleep
@task
def inc(x):
sleep(random.random() / 10)
return x + 1
@task
def dec(x):
sleep(random.random() / 10)
return x - 1
@task
def add(x, y):
sleep(random.random() / 10)
return x + y
@task(name="sum")
def list_sum(arr):
return sum(arr)
with Flow("example-flow-kube") as flow:
incs = inc.map(x=range(4))
decs = dec.map(x=range(4))
adds = add.map(x=incs, y=decs)
total = list_sum(adds)
if __name__ == "__main__":
flow.storage = Docker(registry_url="<>", image_name="<>")
flow.run_config = KubernetesRun(labels=["<>"])
flow.register(project_name="<>")
Javier Domingo Cansino
03/23/2021, 9:03 PMMariia Kerimova
03/24/2021, 12:05 PMJavier Domingo Cansino
03/24/2021, 12:05 PMJavier Domingo Cansino
03/25/2021, 11:27 AMMariia Kerimova
03/25/2021, 12:57 PMJavier Domingo Cansino
03/28/2021, 1:25 PMMariia Kerimova
03/29/2021, 1:36 PMJavier Domingo Cansino
03/30/2021, 1:16 PMJeremiah
task
, so the simplest implementations will package Prefect + execution code together. However, your task can operate however you prefer - for example, inside your Prefect-defined task you could submit work to a remote Dask cluster and wait for its result. Note that the Prefect DaskExecutor
ships the entire Prefect task to the Dask cluster, and therefore does require Prefect to be available; in my example, you would be creating your own Dask client inside the task and using it manually.Javier Domingo Cansino
03/30/2021, 2:30 PMJim Crist-Harif
03/30/2021, 4:45 PM