Hello everyone! Is this a good place to ask for help related to prefect-kubernetes? If it is, this is my issue.

Good morning , our DevOps team set up a Prefect environment in kubernetes. Do I need the API key to be able to connect to the environment or just the PREFECT_API_URL?

For a kubernetes install, how do I do a pip install for prefect_gitlab? Our deveops team installed the environment so I'm wondering if they need to do the install or is there a way for me to do this?

I’m trying to set environment variables in a kubernetes flow run by using the job template but the environment variable value is still using the one that is set when I create my docker Image. How do I override the environment variable set when creating the image with one on a job template to dynamically change how the flow will run?

Oh man gotta knock the dust off these brain cells

Give me a sec to check github

Sorry for the naive question. I'm using prefect-2.7.0 on kubernetes and agent and orion are working as expected. I now have a job that requires additional resources (eg.,

cpu: "4"

and

memory: "16G"

). Where is the recommended place to make that adjustments in prefect to result in the final kubernetes job manifest containing those resource requests? Custom manifest? JSON patch? Is there documentation that I missed somewhere? Thanks for any pointers.

Is there any way with GKE to tag logs with the prefect task id that the log belongs to? I can see flow run ID as part of the logging tabs, but can I get more granular?

Hey all! I'm self-hosting prefect server via kubernetes, and the prefect-hasura component (

<http://docker.io/hasura/graphql-engine:v2.0.9|docker.io/hasura/graphql-engine:v2.0.9>

) has suspiciously long

query_execution_times

in its logs (30-240 seconds) for some POST requests:

{
  "type": "http-log",
  "timestamp": "2022-12-16T18:59:06.914+0000",
  "level": "info",
  "detail": {
    "operation": {
      "query_execution_time": 217.92351778,
      "user_vars": {
        "x-hasura-role": "admin"
      },
      "request_id": "0d820ccb-6294-42f1-8433-a7d1d9562684",
      "response_size": 99,
      "request_mode": "single",
      "request_read_time": 0.000004808
    },
    "request_id": "0d820ccb-6294-42f1-8433-a7d1d9562684",
    "http_info": {
      "status": 200,
      "http_version": "HTTP/1.1",
      "url": "/v1alpha1/graphql",
      "ip": "127.0.0.6",
      "method": "POST",
      "content_encoding": "gzip"
    }
  }
}

and it periodically says it's

gracefully shutting down server

, which restarts the pod. Has anyone here experienced anything like this before?

Hello 👋 in the agent's helm chart, it's currently not possible to specify several queues as the list of queue names will just be concatenated (with a comma in between due to https://github.com/PrefectHQ/prefect-helm/blob/f111427b1cd854e5b733f0d00c47a512a7f93006/charts/prefect-agent/templates/deployment.yaml#L55) Is this known and will it be addressed soon? Thanks 🙏

hi, could someone advise on this issue?

11:57:28.377 | INFO    | prefect.engine - Engine execution of flow run
'ec6916a8-2d40-4ffb-9df3-cfb39f82c875' aborted by orchestrator: This run has
already terminated

No logs anywhere. Tasks/flows show as Crashed.

The orion helm chart refers to "internalPostgres" here, but I can't find this key anywhere; is it real? If not, how should I point my orion at an existing/external postgres server - will it switch automatically if I provide an externalHostName?

Here is a user question about network issue with prefect-helm - sharing here in case somehow knows how to help

I'm new to Prefect. Try to understand how prefect works in the context of k8s. From the documentation, looks like Prefect runs tasks as k8s jobs. Is it possible for Prefect to create any k8s CDRs. For example, if I would like to kick off a Spark run with Spark Operators's SparkApplications. How do I do that with Prefect? TIA for any help.

the path looks wrong to me should be something like "/spec/template/spec/containers/0/resources"

Hello, We have deployed prefect orion on EKS using helm chart and I am trying to start the agent locally on my laptop by setting prefect config set PREFECT_API_URL=*****, but when I try to start the agent using prefect agent start -q **, I am getting this error RuntimeError: The connection pool was closed while 1 HTTP requests/responses were still in-flight. An exception occurred. Any inputs on this are appreciated.

Hi Prefect people! I have a flow that starts around 200 subflows (all a single deployment with different parameters) as kubernetes jobs on a single work queue with a concurrency of 10. I'm running into two issues: First, a few of the subflows never start and are in either a pending or crashed state with either no logs or a single log about downloading flow code. I've increased the timeout window to 10 minutes for pods to start but that doesn't seem to help. Second, some long-running flows have their state changed to crashed with exit status -1 even though they're still running. When they reach their next task, they crash with logs about that task being already finished. Both the agent and the flows are on 2.7.7. Been stuck on this for a few days and would love some advice!

Hi folks! I'm trying to get set up to run an agent & the deployment-flows it manages in an EKS cluster. So far I can start the agent & it is monitoring the expected queue, but it turns out to be running as user

system:serviceaccount:prefect:default

which does not have (and should not have) the necessary permissions. How can I change that to run as a more appropriate user/role?

Hello, In Prefect1 I can specified asked resource with

KubernetesRun

(

cpu_request

,

memory_request

,

cpu_limit

,

memory_limit

) In Prefect2 I use

KubernetesJob

, how can I specify the same parameters?

Deploying my job to kubernetes with

prefect deployment build prefect-test/main.py:main \
    -q test \
    -n test \
    -t api-to-s3 \
    -o deployment.yaml \
    -sb s3/code \
    -ib kubernetes-job/k8s-prod

But I am getting a module not found error. I added

"EXTRA_PIP_PACKAGES": "s3fs"

to my Kubernetes Job block environment so that it can access code on s3 but it feels wrong to add all packages in there. I have a

pyproject.toml

and

requirements.txt

but neither seem to be being used to install packages.

Hi I'm getting submission errors in Kube when trying to run my flows with a custom serviceAccount:

Submission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd3e45036-d667-49d4-8e3b-3a73bec788e1', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '2681d750-c112-4385-86a3-ed13b9c2ebf2', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f3844bc0-2265-4354-9d81-5b753f0d513e', 'Date': 'Mon, 30 Jan 2023 21:34:52 GMT', 'Content-Length': '313'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:data-eng:default\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"data-eng\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}

the deployments:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-agent
  namespace: data-eng
  labels:
    app: prefect-agent
spec:
  selector:
    matchLabels:
      app: prefect-agent
  replicas: 1
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - name: agent
        image: prefecthq/prefect:2.7.8-python3.9
        command: ["prefect", "agent", "start", "-q", "k8s-us-west-2-prod"]
        imagePullPolicy: "IfNotPresent"
        env:
          - name: PREFECT_API_URL
            value: <https://api.prefect.cloud/api/accounts/XX/workspaces/YY>
          - name: PREFECT_API_KEY
            valueFrom:
              secretKeyRef:
                name: prefect-cloud-api-token
                key: prefect-cloud-api-token-value
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  namespace: data-eng
  name: prefect-agent
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/status"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: prefect-agent-role-binding
  namespace: data-eng
subjects:
- kind: ServiceAccount
  name: prefect-agent 
  namespace: data-eng
roleRef:
  kind: Role
  name: prefect-agent
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>

---
# The default manifest generated by `prefect kubernetes manifest agent` uses the default sa in the namespace.
# We will explicitly create a sa and attach the annotation required.

kind: ServiceAccount
metadata:
  name: prefect-agent 
  namespace: data-eng
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::XX:role/an-irsa-role
---

Needles to say that I have specified the same namespace and serviceAccount name in the Blocks but clearly the agent still tries to use the default serviceAccount to create jobs. any idea?

I am using the helm chart to deploy orion and an agent. What would be the recommended way to create blocks and deployments, etc? I don't want to use the UI because I don't want any manual steps. Should it all go through the API?

hi guys, do you have resource/limit recommendation for prefect-orion and agents for example per api request? we are trying to fine tune prefect for k8s.

Hi there

How do you attach secret block in Kubernetes Job and how it can access those secrets during runtime?

Hi folks! I have a flow deployment with type

kubernetes-job

that I want to be able to enable task result caching so all the subflows within the same pod could reuse results. I think I need to set up the pods to have a volume mounted for storage; how can I specify that in the deployment config? Also, there's one task that would be called with the same parameters from multiple concurrent flows. It'd be awesome to be able to cache that result & have it available across pods, but barring that, if I set up caching for that task would it be expected to cache correctly independently for all the pods? 🤞

I am running a flow via kubernetes job. Sometimes when there are not enough nodes available to run the job pods, it will take a sec to scale up a new node. During this time, prefect seems to mark the flow as crashed (since it was not scheduled after some time), but eventually the new node does come up and the flow is able to run fine. However, prefect refuses to run the flow since since the run has already been marked as terminated:

aborted by orchestrator: This run has already terminated.

Is there some way I can configure the internal timeout for waiting for the pod to be scheduled? Configuring retries does not seem to make a difference. Thanks!

Is there anyway to inject Vault secrets to Prefect Kubernetes job automatically without using Blocks?

Hi team, do we use

PREFECT_SERVER_UI_API_URL

or

PREFECT_ORION_UI_API_URL

after the rebranding? Thanks!