https://prefect.io logo
Title
k

kojisuganuma.dog

09/26/2021, 6:55 PM
Hello👋 I’m new to Prefect Cloud. I encountered that my flow was stuck in a
Submitted
state but I cannot find the reason. Any ideas how I can fix this? I’m using Prefect Cloud and Google Kubernetes Engine,
KubernetesAgent
,
KubernetesRun
, GCS storage. Details are in this thread.
1
My Flow
import prefect
from prefect import task, Flow
from prefect.run_configs import KubernetesRun
from prefect.storage.gcs import GCS

from prefect_flows.utils.run_config import make_kubernetes_run_config

@task
def hello_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello world!")

with Flow(FLOW_NAME) as flow:
    hello_task()

flow.run_config = KubernetesRun(
    job_template=make_kubernetes_run_config(
        run_name="example02",
        prefect_cloud_auth_token=<token>,
    ),
)

flow.storage = GCS(
    project=<project>,
    bucket=f"well-{<env>}-prefect-flow-mlops",
    key="flows/example/example02.py",
)
KubernetesAgent’s RBAC https://docs.prefect.io/orchestration/agents/kubernetes.html#rbac Kubernetes Run’s RunConfig
import uuid


def make_kubernetes_run_config(run_name: str, prefect_cloud_auth_token: str) -> str:
  run_id = uuid.uuid4()
  return f"""
apiVersion: batch/v1
kind: Job
metadata:
  name: prefect-job-{run_name}-{run_id}
  namespace: mlops-prefect-agent
  labels:
    app: prefect-job-{run_name}-{run_id}
    identifier: {run_name}-{run_id}
spec:
  template:
    metadata:
      labels:
        app: prefect-job-{run_name}-{run_id}
        identifier: {run_name}-{run_id}
    spec:
      containers:
        - name: flow
          serviceAccountName: prefect-agent
          image: prefecthq/prefect:latest-python3.8
          imagePullPolicy: Always
          command: ["/bin/sh", "-c"]
          args: ["prefect execute flow-run"]
          env:
            - name: PREFECT__CLOUD__API
              value: <https://api.prefect.io>
            - name: PREFECT__CLOUD__AUTH_TOKEN
              value: {prefect_cloud_auth_token}
            - name: PREFECT__CONTEXT__FLOW_RUN_ID
              value: ""
            - name: PREFECT__CONTEXT__NAMESPACE
              value: mlops-prefect-agent
            - name: PREFECT__CLOUD__USE_LOCAL_SECRETS
              value: "false"
            - name: PREFECT__LOGGING__LOG_TO_CLOUD
              value: "true"
            - name: PREFECT__LOGGING__LEVEL
              value: "DEBUG"
            - name: PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS
              value: "prefect.engine.cloud.CloudFlowRunner"
            - name: PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS
              value: "prefect.engine.cloud.CloudTaskRunner"
          resources:
            requests:
              cpu: "200m"
            limits:
              cpu: "200m"
      restartPolicy: Never
      imagePullSecrets:
      - name: ""
"""
KubernetesAgent logs
[2021-09-26 18:36:49,279] INFO - agent | Deploying flow run 1cb8289f-932d-4ae6-823d-a4fdfecacbde to execution environment...
[2021-09-26 18:36:49,606] INFO - agent | Completed deployment of flow run 1cb8289f-932d-4ae6-823d-a4fdfecacbde
INFO:agent:Completed deployment of flow run 1cb8289f-932d-4ae6-823d-a4fdfecacbde
k

Kevin Kho

09/27/2021, 2:03 PM
Hey @kojisuganuma.dog, how often does this happen? Does it happen all the time?
k

kojisuganuma.dog

09/27/2021, 2:39 PM
Yes, it happens all the time.
k

Kevin Kho

09/27/2021, 2:55 PM
It seems there is something wrong with the
make_kubernetes_run_config
function. I think this will be easier if you use the
KubernetesRun()
instead to define these:
RUN_CONFIG = KubernetesRun(image = "prefecthq/prefect:latest-python3.8", service_account_name = "prefect-agent")
But I think there is confusion also. The
agent
is the process that picks up and deploys flows. The RBAC you linked is for the agent (and the task_runner class and cloud API and cloud auth token). These should be for deploying the agent. but not for the Flow. The default Flow job template is very empty .
The agent is attached to a namespace and will deploy flows as jobs in that namespace
k

kojisuganuma.dog

09/27/2021, 3:42 PM
@Kevin Kho It works!!!
The agent is attached to a namespace and will deploy flows as jobs in that namespace
I misunderstood that the namespace where jobs were deployed must be specified in the custom_job_template, but it was wrong! Thank you very much!😊
k

Kevin Kho

09/27/2021, 3:44 PM
Nice! Glad you got it working 🙂
🙌 1