Sorry for the naive question. I'm using prefect-2.7.0 on kubernetes and agent and orion are working as expected. I now have a job that requires additional resources (eg.,

cpu: "4"

and

memory: "16G"

). Where is the recommended place to make that adjustments in prefect to result in the final kubernetes job manifest containing those resource requests? Custom manifest? JSON patch? Is there documentation that I missed somewhere? Thanks for any pointers.

Is there any way with GKE to tag logs with the prefect task id that the log belongs to? I can see flow run ID as part of the logging tabs, but can I get more granular?

Hey all! I'm self-hosting prefect server via kubernetes, and the prefect-hasura component (

<http://docker.io/hasura/graphql-engine:v2.0.9|docker.io/hasura/graphql-engine:v2.0.9>

) has suspiciously long

query_execution_times

in its logs (30-240 seconds) for some POST requests:

{
  "type": "http-log",
  "timestamp": "2022-12-16T18:59:06.914+0000",
  "level": "info",
  "detail": {
    "operation": {
      "query_execution_time": 217.92351778,
      "user_vars": {
        "x-hasura-role": "admin"
      },
      "request_id": "0d820ccb-6294-42f1-8433-a7d1d9562684",
      "response_size": 99,
      "request_mode": "single",
      "request_read_time": 0.000004808
    },
    "request_id": "0d820ccb-6294-42f1-8433-a7d1d9562684",
    "http_info": {
      "status": 200,
      "http_version": "HTTP/1.1",
      "url": "/v1alpha1/graphql",
      "ip": "127.0.0.6",
      "method": "POST",
      "content_encoding": "gzip"
    }
  }
}

and it periodically says it's

gracefully shutting down server

, which restarts the pod. Has anyone here experienced anything like this before?

Hello 👋 in the agent's helm chart, it's currently not possible to specify several queues as the list of queue names will just be concatenated (with a comma in between due to https://github.com/PrefectHQ/prefect-helm/blob/f111427b1cd854e5b733f0d00c47a512a7f93006/charts/prefect-agent/templates/deployment.yaml#L55) Is this known and will it be addressed soon? Thanks 🙏

hi, could someone advise on this issue?

11:57:28.377 | INFO    | prefect.engine - Engine execution of flow run
'ec6916a8-2d40-4ffb-9df3-cfb39f82c875' aborted by orchestrator: This run has
already terminated

No logs anywhere. Tasks/flows show as Crashed.

The orion helm chart refers to "internalPostgres" here, but I can't find this key anywhere; is it real? If not, how should I point my orion at an existing/external postgres server - will it switch automatically if I provide an externalHostName?

Here is a user question about network issue with prefect-helm - sharing here in case somehow knows how to help

I'm new to Prefect. Try to understand how prefect works in the context of k8s. From the documentation, looks like Prefect runs tasks as k8s jobs. Is it possible for Prefect to create any k8s CDRs. For example, if I would like to kick off a Spark run with Spark Operators's SparkApplications. How do I do that with Prefect? TIA for any help.

I’ve been using prefect with some success until today, when I needed to add a memory request on the pods for a specific flow run that the k8s agent is creating. I followed the [example from anna geller](https://github.com/anna-geller/prefect-deployment-patterns/blob/main/blocks/infrastructure_blocks/kubernetes-job/customizations/cpu_memory_limits.py) to add the resource requests to the Kubernetes Job block in the UI. i also added labels to aid in viewing logs in an external application. however, neither the lables nor the customizations made it onto the flow run pods that were created. am i doing something wrong here in defining these blocks? Note that i’m running the

prefecthq/prefect:2.7.7-python3.9

image for the agent

Hello, We have deployed prefect orion on EKS using helm chart and I am trying to start the agent locally on my laptop by setting prefect config set PREFECT_API_URL=*****, but when I try to start the agent using prefect agent start -q **, I am getting this error RuntimeError: The connection pool was closed while 1 HTTP requests/responses were still in-flight. An exception occurred. Any inputs on this are appreciated.

Hi Prefect people! I have a flow that starts around 200 subflows (all a single deployment with different parameters) as kubernetes jobs on a single work queue with a concurrency of 10. I'm running into two issues: First, a few of the subflows never start and are in either a pending or crashed state with either no logs or a single log about downloading flow code. I've increased the timeout window to 10 minutes for pods to start but that doesn't seem to help. Second, some long-running flows have their state changed to crashed with exit status -1 even though they're still running. When they reach their next task, they crash with logs about that task being already finished. Both the agent and the flows are on 2.7.7. Been stuck on this for a few days and would love some advice!

Hi folks! I'm trying to get set up to run an agent & the deployment-flows it manages in an EKS cluster. So far I can start the agent & it is monitoring the expected queue, but it turns out to be running as user

system:serviceaccount:prefect:default

which does not have (and should not have) the necessary permissions. How can I change that to run as a more appropriate user/role?

Hello, In Prefect1 I can specified asked resource with

KubernetesRun

(

cpu_request

,

memory_request

,

cpu_limit

,

memory_limit

) In Prefect2 I use

KubernetesJob

, how can I specify the same parameters?

Deploying my job to kubernetes with

prefect deployment build prefect-test/main.py:main \
    -q test \
    -n test \
    -t api-to-s3 \
    -o deployment.yaml \
    -sb s3/code \
    -ib kubernetes-job/k8s-prod

But I am getting a module not found error. I added

"EXTRA_PIP_PACKAGES": "s3fs"

to my Kubernetes Job block environment so that it can access code on s3 but it feels wrong to add all packages in there. I have a

pyproject.toml

and

requirements.txt

but neither seem to be being used to install packages.

Hi I'm getting submission errors in Kube when trying to run my flows with a custom serviceAccount:

Submission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd3e45036-d667-49d4-8e3b-3a73bec788e1', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '2681d750-c112-4385-86a3-ed13b9c2ebf2', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f3844bc0-2265-4354-9d81-5b753f0d513e', 'Date': 'Mon, 30 Jan 2023 21:34:52 GMT', 'Content-Length': '313'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:data-eng:default\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"data-eng\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}

the deployments:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-agent
  namespace: data-eng
  labels:
    app: prefect-agent
spec:
  selector:
    matchLabels:
      app: prefect-agent
  replicas: 1
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - name: agent
        image: prefecthq/prefect:2.7.8-python3.9
        command: ["prefect", "agent", "start", "-q", "k8s-us-west-2-prod"]
        imagePullPolicy: "IfNotPresent"
        env:
          - name: PREFECT_API_URL
            value: <https://api.prefect.cloud/api/accounts/XX/workspaces/YY>
          - name: PREFECT_API_KEY
            valueFrom:
              secretKeyRef:
                name: prefect-cloud-api-token
                key: prefect-cloud-api-token-value
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  namespace: data-eng
  name: prefect-agent
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/status"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: prefect-agent-role-binding
  namespace: data-eng
subjects:
- kind: ServiceAccount
  name: prefect-agent 
  namespace: data-eng
roleRef:
  kind: Role
  name: prefect-agent
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>

---
# The default manifest generated by `prefect kubernetes manifest agent` uses the default sa in the namespace.
# We will explicitly create a sa and attach the annotation required.

kind: ServiceAccount
metadata:
  name: prefect-agent 
  namespace: data-eng
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::XX:role/an-irsa-role
---

Needles to say that I have specified the same namespace and serviceAccount name in the Blocks but clearly the agent still tries to use the default serviceAccount to create jobs. any idea?

I am using the helm chart to deploy orion and an agent. What would be the recommended way to create blocks and deployments, etc? I don't want to use the UI because I don't want any manual steps. Should it all go through the API?

hi guys, do you have resource/limit recommendation for prefect-orion and agents for example per api request? we are trying to fine tune prefect for k8s.

Hi there

How do you attach secret block in Kubernetes Job and how it can access those secrets during runtime?

Hi folks! I have a flow deployment with type

kubernetes-job

that I want to be able to enable task result caching so all the subflows within the same pod could reuse results. I think I need to set up the pods to have a volume mounted for storage; how can I specify that in the deployment config? Also, there's one task that would be called with the same parameters from multiple concurrent flows. It'd be awesome to be able to cache that result & have it available across pods, but barring that, if I set up caching for that task would it be expected to cache correctly independently for all the pods? 🤞

I am running a flow via kubernetes job. Sometimes when there are not enough nodes available to run the job pods, it will take a sec to scale up a new node. During this time, prefect seems to mark the flow as crashed (since it was not scheduled after some time), but eventually the new node does come up and the flow is able to run fine. However, prefect refuses to run the flow since since the run has already been marked as terminated:

aborted by orchestrator: This run has already terminated.

Is there some way I can configure the internal timeout for waiting for the pod to be scheduled? Configuring retries does not seem to make a difference. Thanks!

Is there anyway to inject Vault secrets to Prefect Kubernetes job automatically without using Blocks?

Hi team, do we use

PREFECT_SERVER_UI_API_URL

or

PREFECT_ORION_UI_API_URL

after the rebranding? Thanks!

Hi everyone, has anyone met this issue before?

ConnectionRefusedError: [Errno 111] Connect call failed (‘127.0.0.1’, 4200)

I am stuck here for over a week. Any guide would be appreciate. Thanks! 😃 https://discourse.prefect.io/t/connectionrefusederror-errno-111-connect-call-failed-127-0-0-1-4200/2415

Hi Everyone, Issue If I create a custom run for a deployment (with k8 job infrastructure) in our prefect cloud account, it creates 2 duplicate jobs in our k8 cluster for the same flow. Not sure what would cause this. Any ideas? Expected Behavior: Only one job should be created per flow fun Extra Info: Prefect Version: Job - 2.7.10 Prefect Version: Agent - 2.8.4 Agent Log

14:16:50.649 | INFO    | prefect.agent - Submitting flow run 'ef043a55-545c-4495-8a63-8263931c06e7'
14:16:51.894 | INFO    | prefect.infrastructure.kubernetes-job - Job 'flow-run-started-for-tx-csv-mock-csv-tpwv2': Pod has status 'Pending'.
14:16:51.897 | INFO    | prefect.agent - Completed submission of flow run 'ef043a55-545c-4495-8a63-8263931c06e7'
14:17:01.677 | INFO    | prefect.agent - Submitting flow run 'ef043a55-545c-4495-8a63-8263931c06e7'
14:17:02.388 | INFO    | prefect.infrastructure.kubernetes-job - Job 'flow-run-started-for-tx-csv-mock-csv-bnhgs': Pod has status 'Pending'.
14:17:02.429 | INFO    | prefect.agent - Completed submission of flow run 'ef043a55-545c-4495-8a63-8263931c06e7'
14:17:26.237 | INFO    | prefect.infrastructure.kubernetes-job - Job 'flow-run-started-for-tx-csv-mock-csv-tpwv2': Pod has status 'Running'.
14:17:29.652 | INFO    | prefect.infrastructure.kubernetes-job - Job 'flow-run-started-for-tx-csv-mock-csv-bnhgs': Pod has status 'Running'.
14:19:44.562 | ERROR   | prefect.infrastructure.kubernetes-job - Job 'flow-run-started-for-tx-csv-mock-csv-tpwv2': Job did not complete within timeout of 5s.
14:19:44.685 | ERROR   | prefect.infrastructure.kubernetes-job - Job 'flow-run-started-for-tx-csv-mock-csv-bnhgs': Job did not complete within timeout of 5s.
14:19:44.719 | INFO    | prefect.agent - Reported flow run 'ef043a55-545c-4495-8a63-8263931c06e7' as crashed: Flow run infrastructure exited with non-zero status code -1.

Hi there, I have various prefect flows which have need for different resources (sidecar container such as cloud_sql_proxy, and different secrets). In Prefect 1, I just created job-template which was shared for all flows. With Prefect Orion, I would like to do it bit differently, and have differentiation based on the flow. Meaning I only configure per flow the actual required resources. Right now I’m thinking of registering different blocks per flow, and just repeating the job-template. Does anyone have any ideas or suggestions how to achieve this with minimal duplication?

Hi everyone, I have a question How to use external Postgres when deploy by Helm Chart? Any guide would be appreciate, thanks!

Hi there, I have a Kubernetes Job block defined with a set of customizations (JSON patches). I am looking to override some of these defaults on a per-deployment basis using infra_overrides. However, it appears that infra_overrides entirely overwrites key/values in the manifest, rather than patching them. This means I cannot keep some customizations and override others. Is this correct, and if so is there another way to apply JSON patches to a k8s job manifest on a per-deployment basis? Thanks in advance for your help!

hello all, iam very new to prefect I deployed prefect-server in kubernetes via helm "https://github.com/PrefectHQ/prefect-helm" it got deployed, now trying to deploy prefect agent but agent pods are crashing due to api key, please find the error message below can any one help me with is? Thnaks in advance

need prefect-api-key how can i get?