Nicolas van de Walle
07/09/2020, 2:55 PMprefect agent install kubernetes -t <MY_TOKEN> -n prefect --label prefect-namespace --rbac | kubectl apply -n prefect -f -
I used Prefect cloud for that. The flow has well been scheduled but nothing gets executed by the agent (it appears in the agents tab in the cloud UI with its prefect-namespace label but it does not see the flows that need to be run).
To be honest, I am quite new to kubernetes and do not really understand how to use RBAC. Do I need to do anything else?Laura Lorenz (she/her)
07/09/2020, 3:07 PM--labels
(which is actually more of a Prefect thing than a kubernetes thing). When you register your flows, do you also associate the prefect-namespace
label to their environment? (The docs on it are here for your ref in case you hadn’t seen them before: https://docs.prefect.io/orchestration/execution/overview.html#labels)Nicolas van de Walle
07/09/2020, 3:22 PMenvironment = DaskKubernetesEnvironment(labels=["prefect-namespace"])
with Flow("Example Flow Without Mail", environment=environment) as flow:
...
But it still does not run the flowLaura Lorenz (she/her)
07/09/2020, 3:30 PMlabels=["prefect-namespace"]
on the environment attached to the flow is exactly what I was looking for, because usually when a flow is stuck in a Scheduled
state even though agents are running it is because the agents and the flow environment don’t have the same labels. If that’s not the case, it must be something else. Is there any output from your agent you can share (this would be in where logs are stored from wherever you deployed your kubernetes agent)Nicolas van de Walle
07/09/2020, 3:32 PM____ __ _ _ _
| _ \ _ __ ___ / _| ___ ___| |_ / \ __ _ ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __| / _ \ / _` |/ _ \ '_ \| __|
| __/| | | __/ _| __/ (__| |_ / ___ \ (_| | __/ | | | |_
|_| |_| \___|_| \___|\___|\__| /_/ \_\__, |\___|_| |_|\__|
|___/
[2020-07-09 14:50:43,735] INFO - agent | Starting KubernetesAgent with labels ['prefect-namespace']
[2020-07-09 14:50:43,736] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-07-09 14:50:43,736] INFO - agent | Agent connecting to the Prefect API at <https://api.prefect.io>
[2020-07-09 14:50:44,019] INFO - agent | Waiting for flow runs...
Laura Lorenz (she/her)
07/09/2020, 3:40 PM"prefect-namespace"
there like we expect, right? In my example here my label is “`Lauras-MacBook-Pro.local`”Nicolas van de Walle
07/09/2020, 3:42 PMimport prefect
from prefect import task, Flow
from random import choice
from prefect.tasks.control_flow.case import case
from prefect.tasks.control_flow.conditional import merge
from time import sleep
from prefect.schedules.schedules import IntervalSchedule
from datetime import timedelta
from prefect import Client
from prefect.environments import LocalEnvironment
from prefect.environments import DaskKubernetesEnvironment
@task
def hello_task():
log('Setting up tasks and waiting for 1s')
sleep(1)
@task
def is_true_task() -> None:
log('Choosing branch')
return choice([True, False]) # Chooses a random boolean value
@task
def first_task() -> int:
log('Executing first_task')
return 1234
@task
def second_task() -> int:
log('Executing second_task')
return 5678
@task
def end_task(x: int):
log('This is the end. Final value is {}'.format(x))
@task
def run_in_parrallel():
log('Running concurrently')
def log(msg: str) -> None:
logger = prefect.context.get("logger")
<http://logger.info|logger.info>(msg)
environment = DaskKubernetesEnvironment(labels=["prefect-namespace"])
with Flow("Example Flow", environment=environment) as flow:
hello = hello_task()
cond = is_true_task()
hello.set_downstream(cond)
with case(cond, True):
val1 = first_task()
with case(cond, False):
val2 = second_task()
parallel = run_in_parrallel()
val = merge(val1, val2, parallel)
end = end_task(val)
flow.register(project_name="Test project 1", labels=["prefect-namespace"])
Laura Lorenz (she/her)
07/09/2020, 3:50 PMNicolas van de Walle
07/09/2020, 3:53 PMLaura Lorenz (she/her)
07/09/2020, 4:03 PMNicolas van de Walle
07/09/2020, 4:27 PMLaura Lorenz (she/her)
07/09/2020, 4:46 PM