has anybody seen this error before?

cannot deploy project with work pool of type 'prefect-agent'

(main) $ prefect deploy flows/modeling.py:foo -n test_deployment -p default-agent-pool

17:00:16.247 | WARNING | ray._private.worker - Failed to set SIGTERM handler, processes mightnot be cleaned up properly on exit.
Cannot deploy project with work pool of type 'prefect-agent'.

Hey @Andy Dienes! It looks like you’re using projects, which is great! Projects can only deploy to typed work pools that are polled by workers. The

default-agent-pool

is a

prefect-agent

work pool that is polled by agents instead of workers, so you can use your project to deploy to a different typed work pool. What infrastructure do you want to run your flows on?

https://github.com/PrefectHQ/prefect-helm/tree/main/charts trying to run on kubernetes

You can see the list of available work pool types in the Prefect UI when creating a work pool. We can also work on adding the list of available work pool types to the documentation. To create a work pool via the CLI, you can run

prefect work-pool create --type kubernetes my-work-pool

and replace

my-work-pool

with the name that you want. This tutorial guides you through the project deployment process and should work with a kubernetes-typed work pool.

│ ValueError: Flow run UUID('71c37c01-e02c-4bff-8d03-fb806545123c') was created from deployment 'tuesdaydepl-5' which is configured with a storage block. Workers currently only support  │
│ local storage. Please use an agent to execute this flow run

I got the kubernetes worker to execute the job, but predictably it cannot find the code. I tried uploading to an s3 storage block but got this error. is using workers with remote storage on the near term horizon? should I just go use an agent? I'd like to be on whatever the most recent recommended pattern is; how are most people doing this?

Workers are designed to be used with projects. Both are currently in beta, but will be the recommended solution once they are out of beta. To retrieve code from S3 with the Kubernetes worker you’ll need to configure

push

and

pull

steps for your project. You can set up a project to use S3 for project storage with

prefect project init --recipe s3

.

I just called

prefect deployment build

on my flow manually; I was getting pretty confused with the project workflow

Those files stay with your code and should be committed to the repository where you’re storing your code. We handle saving the

pull

step remotely so that no modification is necessary for your cluster.

We handle saving the

pull

step remotely

is this just relying on

PREFECT_API_URL

env var to be correct? how do I specify the cluster so that this is saved remotely?

Yes, it will require being able to connect to the Prefect API, but most parts of Prefect require being able to communicate with the API. Connection to your cluster is controlled via the work pool configuration. If you create a Kubernetes Cluster Config block and assign it to your kubernetes work pool, kubernetes workers polling that work pool should be able to connect to your cluster to schedule jobs.

gotcha ok. so if I have multiple clusters I need to change the environment variable each time, there's no like

--address

flag or field in the yamls that can control that

In the

deployment.yaml

file there’s a

job_variables

section that allows overriding default values set on the work pool. To override namespace and cluster config for a deployment would look something like this:

work_pool:
  job_variables:
    cluster_config: "{{ prefect.blocks.kubernetes-cluster-config.my-cluster-config }}"
    namespace: my-namespace

In this case, you’d create a different Kubernetes cluster config block for each cluster that you want to use for flow run execution. You can also use the

-v

flag when running

prefect deploy

to declare overrides.

ah ok, I was just doing

work_pool:
  namespace: my-namespace

it seems the example yaml here

<https://docs.prefect.io/latest/concepts/deployments/>

on "cat facts" has some mention of a field

infra_overrides: {}

but this didn't have the desired effect. seemed like

-v

worked, thanks! I will try putting it under

job_variables

as well