Hello everyone, any idea when the prefect 1.x helm charts will stop being upgraded?

hi folks , I am wondering how to use

on_cancellation

state change hook? From what I've seen so far, if I try to catch

except (asyncio.exceptions.CancelledError, prefect.exceptions.TerminationSignal)

and then return

return Cancelled(_message_=f"Cancelled: {_task_id_} is cancelled")

in the

flow

logic, I ended up with no

cancellation

hook fired at all. I've also tried returning

Cancelling

state, still no go. Could anyone shed some lights on the cancellation hook?

btw, how should I distinguish a

spot instance

shutdown vs. user initiated

cancel

in k8s? Assume I have the ASG with spot instances, when the nodes are being released, I would get

SIGTERM

in the running pods too

Has anyone used GKE autoscaling and run into pod crashes during the autoscaling. I had a flow with 2 pod restart and likely 2 crashes: • restarts ◦ 104759 AM ◦ 105903 AM Looking at the memory spikes after 10:59am I don't see any crashes coinciding with memory spikes. I do see the autoscaler increasing both memory and ram requested though at 10:48am. This coincides with the first crash of 104759am. This appears to be occurring quite a bit and the weird part is the system resources don't appear to be influencing the auto scaling process

Hi all! Getting an issue that Im hoping to get some help on. Im getting this error when running a flow:

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:prefect:prefect-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"prefect\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}

The kubernetes work pool has the namespace set to

prefect

and the service account name set to

prefect-worker

. At first I was able to run jobs and it was working but then I had to make edits to the role to allow it to read secrets and then this started happening. Yaml files I used for the Roles are in the thread! Would really appreciate any insight into this - thanks!

Hi, guys! I am currently deploying a flow with a k8s job block and a docker image. If my flow needs to read/write data to an NFS, how do i do that? I tried mounting a volume in the job manifest but that didn’t work.

Hey all! I’m trying to run a flow that reads and writes to an S3 bucket but I’m struggling with the assignment of the IAM permissions. I’m running my Prefect agent on a K8S pod and as I understand, I need to configuring the pod to use a Kubernetes service account. Is there a way I can add the service account as I running my agent?

This message was deleted.

Is there a way to have a Prefect GPU agent/worker on (temporary) sleep when it’s not used in Kubernetes? Motivation: let’s assume that the cost of a heavy GPU machine is $2k per day. Let’s also assume that our Prefect deployment runs in AKS (managed Azure Kubernetes) and we have separate pods for

prefect-server

and a

prefect-agent

. What if 90% of the day the

prefect-agent

(which is running on a GPU node on the cluster) is idle? This means it’s underutilized and we waste money for no good reason. Reference: Airflow provides Kubernetes Executor - on-demand/ad-hoc worker pods. Since Prefect thought of everything - I’m sure there is either a built-in capability for that or a design pattern for achieving that. Thanks!

Hello all! We're using prefect-ray to run our flow runs on a k8s-based ray cluster. In several places we pass large dataframes into mapped tasks and were running out of object store memory, which we just discovered is an antipattern: https://docs.ray.io/en/latest/ray-core/patterns/pass-large-arg-by-value.html#anti-pattern-passing-the-same-larg[…]value-repeatedly-harms-performance question: is there a way to access the ray instance created by prefect-ray to call ray.put() and ray.get() manually? what's the recommended way to do that?

Hi there, I am having problems with deploying Prefect Streaming self-hosted on Kubernetes. Does Prefect Streaming need to be running on a prefect image or can it be done using a standard Python base image?

Hello everyone, I'am quite new with k8s and I don't understand very how it works with Prefect. I have a k8s cluster and I deployed Prefect server, agent and workers using Helm charts.. My target is to deploy my flows and trigger flow runs using the API. I have all my flows code and it's working locally with prefect server. Now I want to deploy my flows in k8s, how I'am I supposed to do it ? The documentation dont lookss clear to me, is there any "full" code example that I can read 🤔 ?

hi all, we're using k8s on GKE to run our flows. they're triggered from deployments in Prefect Cloud. we're running prefect 2.10.20 with an Agent on k8s. we occasionally see flow runs transition into a CRASHED state before RUNNING and then COMPLETED. the Run Count is 1 in this case. does anyone have suggestions of what we can check? here's an example set of transitions:

2023-08-12T20:00:48.907217+00:00 SCHEDULED Scheduled
2023-08-12T20:00:51.114967+00:00 PENDING Pending
2023-08-12T20:01:52.804776+00:00 CRASHED Crashed
2023-08-12T20:02:05.900502+00:00 RUNNING Running
2023-08-12T20:02:31.613829+00:00 COMPLETED Completed

Hi all! Hoping to get some help with my kubernetes work pool infrastructure. I would like to be able to add a node selector/tolerations to my deployment to ensure that my worker pool and all the jobs that it spins up are on a specific node pool (As this is the only node pool with workload identity enabled and I need it for some flows). Hoping to get some help regarding how I might be able to do this! Thanks!

If i am building a container run as a KubernetesJob, do I need anything outside of the standard prefect package?

I am getting this error at the end of my flow_runs intermittently

ubernetes.client.exceptions.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Audit-Id': '146e8148-7a8d-485c-827e-659f48837db7', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Thu, 24 Aug 2023 16:58:24 GMT', 'Content-Length': '218'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Get \\"<https://10.0.90.47:10250/containerLogs/prefect/gabby-starling-r6fbg-tszq6/prefect-job?follow=true>\\": No agent available","code":500}\n'

Looking for a some help here regarding

OOMKilled

pods of flow runs on k8s. So - putting aside for a moment the reason why it’s OOM to begin with - what happens is that the flow almost completely succeeds (or completely succeeds) - but - somehow still finds itself being run over and over and over to no end. There are no retries defined on the flow itself, and the kube manifest yaml is set to

restartPolicy = never

Yet still, we get this:

Hello everyone ! I'm facing and issue with "Kubernets Jobs" block. In my code I have defined the namespace as follow :

k8s_jb = KubernetesJob(
        ...
        namespace="prefect",
    )

After saving the block I can see in the UI that the namespace is correctly defined. But sometimes, when I run a deployment flow, I get these errors:

TTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:prefect:prefect-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}

The error message indicates that I'm in the wrong namespace. If I use the "Retry" button, at some point it will be executed in the "prefect" namespace. Sometimes I don't get the error, sometimes I have to retry 5 times. Does anyone have the same problem?

How many flows can a single worker handle at a time?

Hi, I’m interested in running Prefect Server in EKS and am looking through this documentation. I’m fairly new to Prefect 2.0 as well as K8s, so thanks in advance for your patience! 😛 I think I understand how to install the server via helm chart, but I’m confused about Workers in terms of “who does the computing”: If I just have a single worker running on a pod, and submit 10 different flows to that worker, will all the flows run on that Worker’s pod? Or will a new pod be spun up for each flow? If I want a new pod to be spun up for each flow, and be terminated when the flow is completed, is there an easy way to specify that?

Hi everyone, I have a issue that has been difficult to debug. We are running prefect 2.8.6 in kubernetes on premise. One node has a gpu with massice memory and CPU, so most of our jobs are run here. What we are experiencing is that when submit a batch of 20 jobs it takes 20 minutes to complete, at the end of the 20 minutes CPU goes through the roof and it start utilizing all the CPU in the node make it unusable.... the jobs hav a 1cpu limit and 10GB memory, 20 jobs should be easy for the node with 112 CPUs and 2TB memory.... any ideas to help us troubleshoot, that would be greatly appreacite it

Hi, I'm running my flow on a kubernetes pod, I have self-hosted agent and worker. Now I see the error logs like this:

Worker 'KubernetesWorker 7763f002-66a8-475f-90cf-b40e048d3e02' submitting flow run '83edf78a-26ec-4f1a-870b-c82b96f25db2'
Creating Kubernetes job...
Job 'axiomatic-oyster-lcgft': Pod has status 'Pending'.
Completed submission of flow run '83edf78a-26ec-4f1a-870b-c82b96f25db2'
Job 'axiomatic-oyster-lcgft': Pod has status 'Running'.
Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/xxxx/workspaces/xxxx/block_schemas/>'
Response: {'detail': 'Workspace scopes missing: manage_blocks'}
For more information check: <https://httpstatuses.com/403>
Job 'axiomatic-oyster-lcgft': Job reached backoff limit.

I masked our account id and workspace id. Anyone has any ideas? Thanks.

Hey all, has any successfully solved the issue where Istio sidecars prevent jobs from finishing properly? I found this thread from a couple years ago, but I was wondering if there's any better ways that I haven't found, since I'm not super keen on forking Prefect for every release, yet we still want to enable Istio sidecars in Prefect jobs for networking security.

Quick and bespoke flow runs run in agent rather than in own animal named pod. I use prefect.yaml to register deployments and flows. Work pools have base job template to make sure namespace and image and whatnot are defined as expected. So why do these quick runs stay inside agent rather than making own new job/pod?

Once I deploy a prefect-worker via helm chart, using value.yaml file, I have only one work-pool mapped to the newly generated worker. Is there any way to assign multiple workpools to a single worker? Here is the values.yaml file I have been using, but this one only activates gke-cpu-5000m-4Gi workpool.

namespaceOverride: prefect
worker:
  cloudApiConfig:
    accountId: HIDE
    workspaceId: HIDE
  config:
    workPool: gke
    workPool: gke-cpu-5000m-4Gi
serviceAccount:
  create: false
  name: "HIDE"

Hi all, I was just wondering if ghere is any template for prefect.yaml file for assigning pre-defined amount of computational resources (including cpu, memory and gpu where the last one is really important for using prefect gcp as ml orchestration tool) in prefect.yaml file. Or should I declare this at the moment of woker creation via helmchart? If so, would the newly created worker in idle would also be assigned to have a gpu? (This looks like a wastage of resource for me).

Hi Since I updated from prefect 2.12.1 to 2.13.0, I have an error when I install the Helm Chart associated to my Prefect version:

Error: parse error at (prefect-worker/charts/common/templates/_labels.tpl:14): unclosed action

Is there a breaking change?

Hello! I'm working on deploying a Prefect worker to a Kubernetes cluster. However, I get

FileNotFoundError

every single time a flow is triggered (tracebacks in the comments). I know for a fact these files/directories exist because I shell into the pods and verify it. Fair to say it's driving me nuts. Here's how I'm set up: 1. Official Prefect Helm chart >> https://github.com/PrefectHQ/prefect-helm/tree/main/charts/prefect-worker a. Nothing modified except the values it asks for. 2. Official Docker image >>

prefecthq/prefect:2.12.0-python3.11-kubernetes

3. I have a

flows

directory with sub-modules for each flow 4. Alongside the

flows

directory, I have two standalone python files that build Blocks using

KubernetesJob()

and

.save()

and deploy Flows using

deployment.Build_from_flow()

. a. In the Dockerfile, I just copy all of this into

/opt/prefect

so resulting directory is

/opt/prefect/flows/, blocks.py, deploy.py

5. I have a custom

entrypoint.sh

script that logs into Prefect, runs the files to build blocks and deploy flows, and then starts the worker. It does not modify file structure at all and everything runs as expected, i.e. Blocks built, Flows deployed, and Worker started in k8s. Any help or pointers as to what is going wrong would be greatly appreciated. I've been trying to debug this for a few days now.

Hi all Anyone that has experience with using the prefect-ray integration? I have a prefect agent and Ray cluster running inside my kubernetes cluster. Both seems works as intended as a standalone. However, when i try to use the prefect-ray integration and specify resources for a task i.e.

from prefect import flow, task
from prefect_ray.task_runners import RayTaskRunner
from prefect_ray.context import remote_options

@task
def process(x):
    return x + 1


@flow(task_runner=RayTaskRunner("ray://<my-ray-service>:10001"))
def my_flow():
    # equivalent to setting @ray.remote(num_cpus=4)
    with remote_options(num_cpus=4):
        process.submit(42)

It fails with a runtime error:

RuntimeError: There is no current event loop in thread 'ray_client_server_1'.

If i do not specify resources for my task, everything works as intended. I have submitted an issue, but it has not been picked up yet, so i thought maybe i could get some help here 🙂 https://github.com/PrefectHQ/prefect/issues/10542

In Prefect 2, Is there a way to get labels that appear in the Kubernetes

job

to appear on the

pod

it generates? Prefect 1.x put useful labels on the

pod

, e.g.

<http://prefect.io/flow_id|prefect.io/flow_id>

, which are no longer present in any Prefect 2

pod

and only appear on the

job

.

job

container labels don’t work for our purposes - we’re trying to use logs to set of alerts, and the label fields have to be on the

pod

. Happy to provide additional information, if it would be helpful!