https://prefect.io logo
Join Slack
Powered by
I know I am being a complete PIA - it will subside...
# ask-community
j
I know I am being a complete PIA - it will subside I promise!! I have the following code in thread - a k8s deployment and a docker deployment - both pointing at exactly the same image - the docker one runs perfectly fine the k8s one has the following error;
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 318, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.9/site-packages/prefect/client/utilities.py", line 40, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/deployments.py", line 197, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 147, in get_directory
    copytree(from_path, local_path, dirs_exist_ok=True)
  File "/usr/local/lib/python3.9/shutil.py", line 566, in copytree
    with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/flows'
Copy code
from prefect.deployments import Deployment
from prefect.infrastructure import KubernetesJob, DockerContainer

from prefect2_agent_repro.repro.flow import repro

infra_k8s = KubernetesJob(
    image="prefect2agentrepro:latest",
    image_pull_policy="Never",
    namespace="prefect",
)

infra_docker = DockerContainer(
    env={"PREFECT_API_URL": "<http://host.docker.internal:4200/api>"},
    image="prefect2agentrepro:latest",
    image_pull_policy="NEVER",
)

deployment_k8s = Deployment.build_from_flow(
    repro,
    name="repro-k8s",
    infrastructure=infra_k8s,
    work_queue_name="default",
    work_pool_name="k8s",
    path="/usr/local/lib/python3.9/site-packages",
)

deployment_docker = Deployment.build_from_flow(
    repro,
    name="repro-docker",
    infrastructure=infra_docker,
    work_queue_name="default",
    work_pool_name="default-agent-pool",
    path="/usr/local/lib/python3.9/site-packages",
)

if __name__ == "__main__":
    deployment_k8s.apply()
    deployment_docker.apply()
Dockerfile...module with flows installed into the container
Copy code
FROM prefecthq/prefect:2.10.12-python3.9

COPY . /opt/src

WORKDIR /opt/src
RUN pip install poetry && poetry build && pip install --verbose ./dist/*.whl

WORKDIR /
n
hi @Joshua Greenhalgh - why is your pull policy never in both cases?
j
Cos I am working locally and it doesn't need to pull anything
the container starts correctly anyway - just fails with the error when it tries to load the flow in the k8s case
minikube cluster
initially while I work stuff out - will be gke eventually though
n
ah ok, was just thinking it was possible that your cluster was using an older image without the new layers where your flow code lives hmm can you explain what's going on with your 
path
setting? why are you pointing at site packages?
j
cos I don't want to manually copy code to a certain location - I am just installing all the flows as a package - was a suggestion somewhere to set the path to site-packages here can't find it now though - I don't want to have to be managing the source and its dependencies separately - pip install the package and get all the deps
To add some complexity to this if I set up an "Agent" as opposed to a "Worker" the flow runs fine - there is some difference somewhere in the logic for looking up and loading the flow src in the worker...
So the agent setup using this manifest;
Copy code
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-agent
  namespace: prefect
  labels:
    app: prefect-agent
spec:
  selector:
    matchLabels:
      app: prefect-agent
  replicas: 1
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
        - name: agent
          image: prefecthq/prefect:2.10.12-python3.9
          command: ["prefect", "agent", "start", "-q", "kubernetes"]
          imagePullPolicy: "IfNotPresent"
          env:
            - name: PREFECT_API_URL
              value: <http://host.docker.internal:4200/api>
            - name: PREFECT_API_KEY
              value: None
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: prefect-agent
  namespace: prefect
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "pods/status"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: prefect-agent-role-binding
  namespace: prefect
subjects:
  - kind: ServiceAccount
    name: default
    namespace: prefect
roleRef:
  kind: Role
  name: prefect-agent
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
  name: prefect-agent
rules:
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRoleBinding
metadata:
  name: prefect-agent-cluster-role-binding
subjects:
  - kind: ServiceAccount
    name: default
    namespace: prefect
roleRef:
  kind: ClusterRole
  name: prefect-agent
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
and the deployment looks like;
Copy code
infra_k8s = KubernetesJob(
    image="prefect2agentrepro:latest",
    image_pull_policy="Never",
    namespace="prefect",
)

deployment_k8s_agent = Deployment.build_from_flow(
    repro,
    name="repro-k8s-agent",
    infrastructure=infra_k8s,
    infra_overrides={"namespace": "prefect"},
    work_queue_name="kubernetes",
    path="/usr/local/lib/python3.9/site-packages",
)
Open in Slack
PreviousNext
Powered by