Does Prefect 2 have something analogous to

wait_for_flow_run

from Prefect 1? I'd like to have a flow of flows which stops in a Failed state if one of its subflows fails, but I'm not sure how best to implement this in Prefect 2.

Hi, do you think prefect (2) server will scale up to thousand of deployment with scheduled each 30s ?

other question, to be honest my project is a uptime kuma like : should I create one deployement for all probes (where each probe is a website as example) or as much deployment than website to probe ?

I have two flows deployed on two distinct aws accounts that I'd like to coordinate as subflows. Right now I'm calling them sequentially with something like

my_flow_run = prefect.deployments.run_deployment(name='my_deployment')

, then throwing an error if

my_flow_run.state_name=='failed'

. I'd like to coordinate them asyncronously, with something like

my_flow_run_1, my_flow_run_2 = await asyncio.gather(
        run_deployment(
            name=f"my_deployment_1"),
        run_deployment(
            name=f"my_deployment_2")
    )
throw_error_if_flow_run_failed(my_flow_run_1)
throw_error_if_flow_run_failed(my_flow_run_2)

then I'd like to continue the rest of the main flow sequentially. Am I going about this the right way? is there a built-in solution or will I need to hand-roll something?

Hi everyone, Does prefect gives us a better clue of what's happening when a job goes to Late status? I have an ECS cluster and task ready and all set up with ecs block still when I run from UI , I get this (screenshot attached)! I wish there was a better logging here.

hey all, i am trying to update the

flow_run_name

in prefect 1. looks like the flow run name gets updated in the client in

RenameFlowRun.run

, but the prefect context doesn't. is there an easy way to update the prefect context's

flow_run_name

?

hi a quick question for prefect 2, I am planning to deploy 2 prefect agent (they have the same flows, but running on different python package) to the server, should I be using a different work pool or work queue? **not intending to duplicate the flow runs, just that it is easier to have the flows in both agents for deployments

Has anyone ever documented (or can share) a release process for prefect flows code? Say, I have a repo that describes a few flows: A, B, and C. I have the code deployed as a docker image + deployment in GCS bucket. I would now like to allow gradual/controlled rollout of the repo code changes (deployment code to GCS, and container image) to each of A, B, C in each of QA and Prod environments. The not-the-most-straightforward way I can think of is: 1. For every commit in the repo, push a docker image with the commit hash as a label and deploy code to e.g.

gs://<bucket>/<commit hash>

2. For each of the cartesian product of Flow ⅹ Env (e.g.

A-prod

,

B-qa

) have: a. A separate storage block pointing to a different subdir of

gs://<bucket>/

b. A docker image label pointing to the image of the same commit. 3. The release to an environment would then include: updating the storage block to point to a different dir + updating the container image tag. Each deployment would also have an

infra_override

pointing it at its specific image tag. A couple of concerns I’m having are: storing all code/image versions (might be a good thing?). Maintaining potentially a large number of storage blocks (one for each of flow x env). Any thoughts?

Hello everyone, I'm getting this error when I run the scheduled prefect job today with ECS Service. the only difference compared to yesterday's successful run was that I needed to recreate the image and replace the old ECR image (kept the repository). The ECR repository is private but ECS task Auto-assign public IP is

enabled

so this error shouldn't happen as per AWS I don't need to set up NAT gateway to allow my subnets to talk to ECR through the internet, or have to setup ECR VPC Endpoints. Also, security group inbound and outbound is set for all ports and there is no secret manager used.

Submission failed. prefect_aws.ecs.TaskFailedToStart: ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post "<https://api.ecr.us-west-2.amazonaws.com/>": dial tcp 34.223.27.182:443: i/o timeout. Please check your task network configuration. See Less

I’m having an issue with running flows on kubernetes (minikube). I have a server and an agent running in minikube. I’m using gcs as storage and kubernetes job as infrastructure. I have a file in my repo called

flows/api_flow.py

and the entrypoint is

./flows/api_flow.py:call_api

. The error comes when triggering the flow to run it on kubernetes. Failes with this error:

prefect.exceptions.ScriptError: Script at 'flows/api_flow.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')

how can two projects share code? I have some

utils

I want to be common to all projects, but i want to develop and deploy

projA

and

projB

separately. I'm using the s3 recipe

hello, need help on understanding/troubleshooting: I have just deployed a new agent. call it agent b on top of agent a (current agent) to my prefect server (version 2) (self hosted) all on rancher. I am able to deploy the flow in agent b cli, and it appears in the ui of prefect server, however when i trigger the run, it says

Flow could not be retrieved from deployment. FileNotFoundError: [Errno 2] No such file or directory

I am suspecting it is not looking in the right agent. My questions: 1. How does prefect server knows which agent to search from when a run happens? 2. How to troubleshoot to check if a flow is trying to get the code from the right agent?

I am trying to wrap my head around the correct way for projects and blocks to interact now that projects are a thing. I am wondering if anyone can shed some light on this for me. I will use our infrastructure as an example. Most of our deployments run on Google Cloud Run. Before, it made a lot of sense to create a

Cloud Run Job

block, save it, and reuse if in building our python

Deployment

objects. Now, it seems that a lot of the infrastructure is taken over by the work pool. If I create a

cloud run work pool

, what is the purpose of a

cloud run block

? I already have all of my infrastructure existing in code now and I don't really see why I should want to use a

cloud run block

. Other things like credentials still make a lot of sense for blocks. But some of them seem to be replaced by the new projects + work pools workflow. Am I missing something here? Or does anyone have some good insights into this they would be willing to share?

I'm curious about a bump to this question from a week or two ago: https://prefect-community.slack.com/archives/CL09KU1K7/p1682622478391679

Hello, I used to able to build and deploy the same exact ways a few days ago and now looks like prefect and docker has connection issue. I also tested scripts on terminal to work with docker desktop and everything is fine and also gave permission to /var/run/docker.sock and added my user to docker group. All agents also running fine. Please advise. _prefect deployment build flows/python/build_file.py:load_credit_sf -n prefect-docker-creditfurnishing -q docker -sb s3/prefect-s3/credit-furnishing/docker-local-image -ib docker-container/prefect-docker-creditfurnishing --apply_

_retrieve_server_version raise DockerException( docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')) The above exception was the direct cause of the following exception: RuntimeError: Could not connect to Docker.

Hello, We are trying to deploy changes from a monorepo that have multiple projects where our code lives. The structure looks something like this:

python_repo /
   projectA / 
      src/projectA/flows/...
      pyproject.toml
   projectB /
      src/projectB/...
      pyproject.toml

projectA depends on projectB and I’d like to deploy all live changes within the python_repo. What is the recommended approach for doing this?

Hello! I'm creating

Deployment

from Python code, but I need to set a custom

PREFECT_API_URL

but not through an environment variable. Is there a way to update the

Settings

directly from code?

Hello, I am using prefect==2.6.5 on production with work queues as systemd services. I want to migrate to at least 2.10. Please provide a logical setup for work pool and queues. I don't understand what the pools feature is trying to solve. A tutorial/guidance in the work pool and queue operate together will surely help.

Hi. Is there any guidance for using projects and deploying to use aws batch instead of ECS.

Hello, With prefect in production over 50 million records of logs have been generated. Now prefect server is not able to fetch the records. Please help. Screenshot of the feedback is as in the image and am certain the task has logs.

Hi All. Is there any best practices documentation on how we create multiple deployments in Prefect for different flows? Like do we need to create different deployment.py file or different YAML files or single YAML file?

hi for https://docs.prefect.io/2.10.21/guides/cloud-run-push-work-pools/ that is just announced, does it work for self hosted prefect server?

Any best practices for how to profile a prefect flow effectively? Profilers generally don't seem to like Prefect that much. Any easy way to run a flow as a function outside of the prefect engine and just a normal python function so it is easier to profile?

We do have two problems with Prefect flow when using Prefect-Databricks: • the job_sumit_and_wait flows uses subflows that ends up polluting the UI (who cares about how many times you polled to get the result from Databricks?) • the information to find the Databricks job are not located in our parent job that really submit the job on Databricks, but we need to navigate to child flos

hey folks, any update on when prefect 2 will be compatible with pydantic 2?

Hi everyone, not sure if this is the correct channel to post this. I'm using ECS Task as infrastructure, it works fine except for the following: The flow state enters a

crashed

state for a minute or so and then I changes to

running

. Is that the expected behaviour?

I’m curious what is the upside using prefect cli with prefect.yaml to build my image instead of building it using, for example, github action?

How does everyone split their workflows for environments if using same Prefect account? Eg Prod, Stage, Dev...

Hi i am planning to do a cicd flow deployment in our own self hosted cicd which are on the same network as the self hosted prefer server 2.0. For flows deployment to work in cicd, is it right to say that as long as i have the env var PREFECT_API_URL set in cicd run, the flow deployment ie.

deployment.build()

and

deployment.apply

would be registered in the prefect server even if the cicd container is not going to be running the flow(not a long time running prefect agent)?

Hi! I've tried the Azure Container Instance push work pool and it seems to work nicely. It takes about 1min30 to create the deployment on Azure, is there a way to speed this up? I'm using the official prefect docker image + a few additional python packages