Hi! i'm trying to log the duration of each flow/task without adding too much code. I see the duration shows up in the UI so it's calculated somewhere, any easy/hacky way to access the start/end times of each task and flow?

Hi folks -- I think I know the answer to this, but is it currently unsupported to run Prefect Server behind a reverse proxy with a prefixed path -- e.g. https://example.com/prefect ? I see a number of bugs and some proposed MRs, but looks like everything has been closed as "won't fix" / "too complicated" and, indeed, I seem unable to get this working locally.

hey all! Is there an easy way to prevent one deployment from having multiple runs? So if it tries to run and it is already running, it will not start the second one?

Hey folks, we've been running into an issue where our Prefect Worker memory usage is constantly increasing until it get to the point where we run out of memory and the service restarts, has anyone experienced this issue or know what is causing this? To give a bit of background, we run our Prefect Worker in a self hosted Kubernetes cluster and it was installed via Helm. Originally we used the default memory limit of 1GB but have since increased to 4GB. The worker is on version 2.11.2

CONCURRENCY GAPS: i have 1000 tasks. concurrency tag set to 20. it runs 20 concurrently then there is this 20-30 second gap before it does another 20 tasks. how do i prevent this time delay “gap” and make it just keep running 20 constantly until done. where is this delay coming from? hard to see in flow screenshot but the little green blobs are the intermittent 20 tasks being completed.

Hi all, I have issues with deleting a flow from the Orion. Does anybody know the fix?

Hello, since the flow has to be unique within a workspace, I would like to alert the developer during PR if they’re trying to use a name that already exists. Are there any apis to validate this using the prefect. yaml . I don’t want a surprise when i run prefect deploy command Any suggestions ?

Hi folks, did someone have same issue or can help me to solve it? I have a flow that run 2 dbt projects one after another as subflows in one flow. The storage of flows and dbt projects is GitHub. Prefect Agent is running on Azure VM. First dbt flow runs always successfully while second flow sometimes has this error and I can’t find the reason why. Any ideas? Thanks a lot!

Finished in state Failed('Flow run encountered an exception. shutil.Error: [(\'C:\\\\Users\\\\user_x\\\\AppData\\\\Local\\\\Temp\\\\tmpu8dj3ri0prefect\\\\.git\\\\objects\\\\pack\\\\pack-dcb61de38801bc82e121d9ddd04df4adace353d8.idx\', \'C:\\\\Users\\\\user_x\\\\AppData\\\\Local\\\\Temp\\\\tmpj3paxn4kprefect\\\\.git\\\\objects\\\\pack\\\\pack-dcb61de38801bc82e121d9ddd04df4adace353d8.idx\', "[Errno 13] Permission denied: \'C:\\\\\\\\Users\\\\\\\\user_x\\\\\\\\AppData\\\\\\\\Local\\\\\\\\Temp\\\\\\\\tmpj3paxn4kprefect\\\\\\\\.git\\\\\\\\objects\\\\\\\\pack\\\\\\\\pack-dcb61de38801bc82e121d9ddd04df4adace353d8.idx\'"), (\'C:\\\\Users\\\\user_x\\\\AppData\\\\Local\\\\Temp\\\\tmpu8dj3ri0prefect\\\\.git\\\\objects\\\\pack\\\\pack-dcb61de38801bc82e121d9ddd04df4adace353d8.pack\', \'C:\\\\Users\\\\user_x\\\\AppData\\\\Local\\\\Temp\\\\tmpj3paxn4kprefect\\\\.git\\\\objects\\\\pack\\\\pack-dcb61de38801bc82e121d9ddd04df4adace353d8.pack\', "[Errno 13] Permission denied: \'C:\\\\\\\\Users\\\\\\\\user_x\\\\\\\\AppData\\\\\\\\Local\\\\\\\\Temp\\\\\\\\tmpj3paxn4kprefect\\\\\\\\.git\\\\\\\\objects\\\\\\\\pack\\\\\\\\pack-dcb61de38801bc82e121d9ddd04df4adace353d8.pack\'")]\n')

Hey Community, I am trying to use Marvin in a project, but I can see that prefect 2.0 does not like

pydantic > 2.0

and marvin requires

>= 2.0

Any chance this will be getting resolved any time soon?

This message was deleted.

I am deploying my Prefect flows in Docker. I have the deployment deploying correctly, and when I create a job it is being picked up by a worker and running in Docker correctly. Two questions • How do I change the docker networking to "host" for the running container (I want to be able to access a service locally on the same host)? • How do I have the container delete itself after a run?

Hi! I am using the work pool Azure Container Instance:push. For specific deployments which uses this work-pool, how can I override the docker image url?

Hello! I am currently running into issues when using @tasks in a flow, we are iterating through a lot of dates and pass very large pandas dataframes to a task for each date, what I notice is that when you do this after about the 10th day our flow gets OOMKilled, if we don't use tasks it doesn't seem to have an issue. The problem being our upsert method is currently a task. My assumption is that the flow is keeping track of the parameters sent to each task and effectively running out of memory by trying to keep each of our dataframes in memory. Is there a way to avoid this or any information on this kind of thing? 🙂

Hello, Is there a way to make the flow end in a failed state if any logger.error was printed or if any exception was raised?

Is there a good example of how to do subflows on your local account & on a local server? The documentation is unclear about what you would do if you had 1 flow in Python File 1 and 1 in Python File 2 and you wanted to schedule them as subflows together in Python File 3.

Prefect 2.0 migration newbe question ... I created a Organization, with one workspace (added billing and paid with credit card for new users) but when I invite other team members it asks them that there is no organization and asks them to create a space

I am new here and working on understanding migration from Prefect 1 to 2. Is Prefect 1 officially deprecated or just not receiving support from the team anymore?

I'm trying to figure out how to use the prefect cli to get secrets

marvin told me this

but .. he lies..

any thoughts?

Hi 👋 We’ve started seeing some Crash states of our runs.

In the lat couple hours we have started receiving 404s back from api.prefect.cloud when attempting to call flow_runs -- is there something down? it doesnt appear to be related to AWS issues seen today

Creating work pool through CI/CD process: We are using PrefectCloud with Worker Running on ECS Task, and based on the configuration in Deployment it is creating Both flows and infra is written in AWS CDK in the same mono repo and we have plan to deploy this through git hub actions. I understand I can create work-pool through prefect work-pool create CLI however Is it possible to run this through CI/CD where it will only create if the work pool does not exist or update it there is any changes to the work-pool? Currently if I use prefect work-pool name it would then throw error if the work-pool exist. While I could use prefect work-pool ls then read the output programatically and only create if this does not exist, are there any easier way I am missing I should consider when if I want to create work-pool also automatically part of build and deployment? Or possibly I am missing something basic here? Or keen to know if people Automate creation of work pool or that is something one off created through UI, as flow usually changes more compare to work pool on prefect cloud.

hi ! i have been using prefect 2.7.8 version for a very long time . we are trying to upgrade to newer version of prefect to explore newer features. was bit confused by the difference of workers and agents . also when i self hosted prefect server ui (like before) the blocks catalog was empty.. any particular reason for this ? shall i just create them using prefect cli or python sdk , (creating from block catalog was easier )

Bonjorno friends, I have been trying to make the prefect cloud work for the past few days. After resolving various errors and deployment crashes, I encountered a new issue: the agent is raising a NotImplementedError. To address this, I followed the suggestion in an article I found and updated to Python 3.11.5. However, I am now encountering the following error:

(anton) C:\Users\Anton\Desktop\DE2\2_workflow_orchestration\flows\03_deployments>python flow_pandas.py  
Traceback (most recent call last):
  File "C:\Users\Anton\Desktop\DE2\2_workflow_orchestration\flows\03_deployments\flow_pandas.py", line 5, in <module>
    from prefect import flow, task
  File "C:\Users\Anton\anaconda3\envs\anton\Lib\site-packages\prefect\_init_.py", line 37, in <module>
    from prefect.runner import Runner, serve
  File "C:\Users\Anton\anaconda3\envs\anton\Lib\site-packages\prefect\runner.py", line 46, in <module>
    from prefect.client.orchestration import get_client
  File "C:\Users\Anton\anaconda3\envs\anton\Lib\site-packages\prefect\client\orchestration.py", line 11, in <module>
    import pydantic
  File "pydantic\_init_.py", line 2, in init pydantic._init_
  File "pydantic\dataclasses.py", line 41, in init pydantic.dataclasses
    # +=========+=========================================+

ModuleNotFoundError: No module named 'typing_extensions'

I updated all the packages in my env, prefect, python, etc. checked all the possible solutions around 'typing_extensions'. Maybe I missed something obvious? Do you have any suggestions to where should I look? Cheers!

Would Prefect (cloud) be a good choice for a "queue of async logic" type of workload? It's not a typical scheduled data ETL pipeline scenario but instead it's small parts of logic that are triggered by actions of users of your app. For example, in our app, when a user purchases a coupon we want to start a flow that performs some logic (some DB and API calls) in a durable fashion. Specifically, we would use the ECS push work pools. So let's say tens, hundreds of new ECS tasks per minute. I'm wondering if this is a viable use case of Prefect (performance and cost wise), compared to something more traditional like, let's say, a kafka topic with a consumer that performs this logic asynchronously? Thanks!

Hello again! 'm relatively new to working with Prefect, and my current project involves integrating it into my Kedro pipeline. Unfortunately, I've run into some challenges when attempting to run it within a Docker environment. I've been struggling to locate any documentation that specifically addresses the integration of Prefect into a Kedro pipeline project. It seems that implementing Prefect within a Kedro pipeline may require a distinct approach compared to using it in regular Python scripts. I would greatly appreciate any guidance or insights you can provide. Thank you in advance! Also, I'd like to share an additional experience as well. In the past, I faced connectivity problems with Prefect, even though I had started the Prefect agent. I was unable to establish a connection to both the local server and cloud server instances. To resolve this issue, I resorted to starting the entire process from scratch. I'm wondering if there might have been a specific step I missed or if there's something I may have done incorrectly that led to this problem. Your insights on this matter would be greatly appreciated. Thank you once again!

I am working on a Custom Worker and am getting this error in my flow logs

Infrastructure returned without reporting flow run 'ae92eb90-d347-4c4d-b15b-ad4cbf3b5466' as started or raising an error. This behavior is not expected and generally indicates improper implementation of infrastructure. The flow run will not be marked as failed, but an issue may have occurred.

I am returning a 0 status_code and an identifier, so I can't figure out what is triggering this