heyy, I am setting up prefect in an on-prem production environment and I'm looking into setting up Storage. I would like to know if there is any recommended KV store docker image that I should use for this purpose?

Hey guys, running into this issue with prefect, and I'd appreciate some insight. I'm working on just the tutorial from the docs, and the page isn't loading

I am trying to transition our deployments from a Cloud Run worker pool to a Cloud Run V2 worker pool in order to leverage the longer timeouts >3600s. I have been doing testing but have had issues getting it to work. Any time I run a deployment on the worker pool I get the following error:

Flow run could not be submitted to infrastructure: <https://run.googleapis.com/v2/projects/trading-system-dev/locations/us-central1/jobs?jobId=prefect-cyber-sparrow&alt=json> returned "Violation in CreateJobRequest.job.template.template.vpc_access.connector: one and only one of connector and network_interfaces must be set.". Details: "[{'@type': '<http://type.googleapis.com/google.rpc.BadRequest|type.googleapis.com/google.rpc.BadRequest>', 'fieldViolations': [{'field': 'Violation in CreateJobRequest.job.template.template.vpc_access.connector', 'description': 'one and only one of connector and network_interfaces must be set.'}]}]">

From what i've gathered, prefect uses Terraform and cloud run seems to require either a connector or network_interfaces. I am not specifying a vpc connector (which prefect has marked as optional). It almost seems as though when there is no vpc connector, the network_interfaces should be used as a default but is not happening. Any help or advice would be greatly appreciated.

is there anyway to set the

max_retries

on a task from the UI (or without having to upload new code)? I was hoping I could template the value and pass it to the @task decorator, but it doesn't look like you can use templates with the

max_retries

argument

Hello everyone!! I am having some issues getting the UI to run correctly on a Kubernetes cluster, deployed with the Prefect Helm chart I am using the L7 ingress for apollo and the UI and I keep getting this error: I am able to register projects and flows with the CLI, so I think everything is connecting except for the UI. Here's my Helm chart: Any ideas on where I might be going wrong?

@Marvin how to trigger a flow with api webhook in prefect server?

@Marvin I am trying to create a prefect.yaml file using prefect version 2.x I get this error when I try to use the command prefect deploy in the terminal: You've chosen a work pool with type 'prefect-agent' which cannot be used for project-style deployments. Let's pick another work pool to deploy to. My deployment section of the prefect.yaml file looks like this: deployments: - name: introduction.template.test_flow version: null tags: [] description: null schedule: {} flow_name: null entrypoint: template\test_flow\flow.py:test_new_deploy_flow parameters: {} work_pool: name: default-agent-pool work_queue_name: prod-limit-1 job_variables: image: '{{ build_image.image }}' How do I create a prefect.yaml file that can deploy to a work pool with type prefect-agent?

What’s the best practice for Data Retention Policy on Prefect deployment runs? Just as a reference, here is how it is implemented for Apache Airflow, as yet another garbage collector DAG: https://stackoverflow.com/questions/66580751/configure-logging-retention-policy-for-apache-airflow I’m sure that Prefect has either a built-in mechanism for that, or encourages a common idiom for rotating / archiving / deleting artifacts from old runs. Context: We have a persistent storage on Azure Blob Storage (the S3 equivalent) where we store artifacts (e.g. output files and images) from a Machine Learning (Kedro) run. The space can pile up pretty quickly across runs and we would run out of storage, rendering our Prefect deployments not operational. What kind of policies are recommended to evict data from old runs? I don’t want to run out of space and I want the Prefect pipelines to remain operational. I know that some of you would say: “_It depends_”, so for the sake of this example let’s imagine that I have a dedicated 256GB of storage. Should I set a threshold (e.g. 70% of full) that will be as a trigger for evicting (removing) artifacts from old runs? Also, when should this run? as the first (prerequisite) subflow in my bigger flow, or as yet another deployment in Prefect on a recurring schedule? Thanks!

Is this heartbeat configuration only applicable to Prefect cloud setups? https://docs.prefect.io/orchestration/concepts/services.html#heartbeat-configuration I have a (expected) long running job on a Prefect server setup trying to not have the zombie killer terminate the job:

No heartbeat detected from the remote task; marking the run as failed.

Hi all, is there a preferred way to pass along secrets / env vars to the helm deployment of prefect-server during a CI/CD pipeline? For more context, I'm trying to give the deployment a

AZURE_STORAGE_CONNECTION_STRING

for access to blob storage. I'm not seeing anything baked into the helm values.yaml (though maybe something in

prefectConfig

could work?). I was thinking of attaching a k8s secret after the helm deployment? But I think it needs to be set on the job that the agent kicks off. And I should be able to set the env var as-is, or in secret form like

PREFECT__CONTEXT__SECRETS__AZURE_STORAGE_CONNECTION_STRING

?