Andy Dienes
04/10/2023, 5:49 PMcannot deploy project with work pool of type 'prefect-agent'
(main) $ prefect deploy flows/modeling.py:foo -n test_deployment -p default-agent-pool
17:00:16.247 | WARNING | ray._private.worker - Failed to set SIGTERM handler, processes mightnot be cleaned up properly on exit.
Cannot deploy project with work pool of type 'prefect-agent'.
alex
04/10/2023, 5:53 PMdefault-agent-pool
is a prefect-agent
work pool that is polled by agents instead of workers, so you can use your project to deploy to a different typed work pool. What infrastructure do you want to run your flows on?Andy Dienes
04/10/2023, 9:41 PMalex
04/11/2023, 1:37 PMprefect work-pool create --type kubernetes my-work-pool
and replace my-work-pool
with the name that you want. This tutorial guides you through the project deployment process and should work with a kubernetes-typed work pool.Andy Dienes
04/11/2023, 3:45 PM│ ValueError: Flow run UUID('71c37c01-e02c-4bff-8d03-fb806545123c') was created from deployment 'tuesdaydepl-5' which is configured with a storage block. Workers currently only support │
│ local storage. Please use an agent to execute this flow run
I got the kubernetes worker to execute the job, but predictably it cannot find the code. I tried uploading to an s3 storage block but got this error. is using workers with remote storage on the near term horizon? should I just go use an agent? I'd like to be on whatever the most recent recommended pattern is; how are most people doing this?alex
04/11/2023, 3:49 PMpush
and pull
steps for your project. You can set up a project to use S3 for project storage with prefect project init --recipe s3
.Andy Dienes
04/11/2023, 3:51 PMprefect deployment build
on my flow manually; I was getting pretty confused with the project workflowprefect project init --recipe s3
but it's not clear to me where those files should live (on my local? on the server cluster?) and how to keep them in syncalex
04/11/2023, 3:56 PMpull
step remotely so that no modification is necessary for your cluster.Andy Dienes
04/11/2023, 3:57 PMWe handle saving theis this just relying onstep remotelypull
PREFECT_API_URL
env var to be correct? how do I specify the cluster so that this is saved remotely?alex
04/11/2023, 4:00 PMAndy Dienes
04/11/2023, 4:01 PM--address
flag or field in the yamls that can control that--override namespace=mynamespace
in previous deployment builds. how do I do that with projects?alex
04/11/2023, 4:08 PMdeployment.yaml
file there’s a job_variables
section that allows overriding default values set on the work pool.
To override namespace and cluster config for a deployment would look something like this:
work_pool:
job_variables:
cluster_config: "{{ prefect.blocks.kubernetes-cluster-config.my-cluster-config }}"
namespace: my-namespace
In this case, you’d create a different Kubernetes cluster config block for each cluster that you want to use for flow run execution.
You can also use the -v
flag when running prefect deploy
to declare overrides.Andy Dienes
04/11/2023, 4:14 PMwork_pool:
namespace: my-namespace
it seems the example yaml here <https://docs.prefect.io/latest/concepts/deployments/>
on "cat facts" has some mention of a field
infra_overrides: {}
but this didn't have the desired effect. seemed like -v
worked, thanks! I will try putting it under job_variables
as well