Hi all, I have issues with deleting a flow from the Orion. Does anybody know the fix?

Hello, since the flow has to be unique within a workspace, I would like to alert the developer during PR if they’re trying to use a name that already exists. Are there any apis to validate this using the prefect. yaml . I don’t want a surprise when i run prefect deploy command Any suggestions ?

Hi folks, did someone have same issue or can help me to solve it? I have a flow that run 2 dbt projects one after another as subflows in one flow. The storage of flows and dbt projects is GitHub. Prefect Agent is running on Azure VM. First dbt flow runs always successfully while second flow sometimes has this error and I can’t find the reason why. Any ideas? Thanks a lot!

Finished in state Failed('Flow run encountered an exception. shutil.Error: [(\'C:\\\\Users\\\\user_x\\\\AppData\\\\Local\\\\Temp\\\\tmpu8dj3ri0prefect\\\\.git\\\\objects\\\\pack\\\\pack-dcb61de38801bc82e121d9ddd04df4adace353d8.idx\', \'C:\\\\Users\\\\user_x\\\\AppData\\\\Local\\\\Temp\\\\tmpj3paxn4kprefect\\\\.git\\\\objects\\\\pack\\\\pack-dcb61de38801bc82e121d9ddd04df4adace353d8.idx\', "[Errno 13] Permission denied: \'C:\\\\\\\\Users\\\\\\\\user_x\\\\\\\\AppData\\\\\\\\Local\\\\\\\\Temp\\\\\\\\tmpj3paxn4kprefect\\\\\\\\.git\\\\\\\\objects\\\\\\\\pack\\\\\\\\pack-dcb61de38801bc82e121d9ddd04df4adace353d8.idx\'"), (\'C:\\\\Users\\\\user_x\\\\AppData\\\\Local\\\\Temp\\\\tmpu8dj3ri0prefect\\\\.git\\\\objects\\\\pack\\\\pack-dcb61de38801bc82e121d9ddd04df4adace353d8.pack\', \'C:\\\\Users\\\\user_x\\\\AppData\\\\Local\\\\Temp\\\\tmpj3paxn4kprefect\\\\.git\\\\objects\\\\pack\\\\pack-dcb61de38801bc82e121d9ddd04df4adace353d8.pack\', "[Errno 13] Permission denied: \'C:\\\\\\\\Users\\\\\\\\user_x\\\\\\\\AppData\\\\\\\\Local\\\\\\\\Temp\\\\\\\\tmpj3paxn4kprefect\\\\\\\\.git\\\\\\\\objects\\\\\\\\pack\\\\\\\\pack-dcb61de38801bc82e121d9ddd04df4adace353d8.pack\'")]\n')

Hey Community, I am trying to use Marvin in a project, but I can see that prefect 2.0 does not like

pydantic > 2.0

and marvin requires

>= 2.0

Any chance this will be getting resolved any time soon?

This message was deleted.

I am deploying my Prefect flows in Docker. I have the deployment deploying correctly, and when I create a job it is being picked up by a worker and running in Docker correctly. Two questions • How do I change the docker networking to "host" for the running container (I want to be able to access a service locally on the same host)? • How do I have the container delete itself after a run?

Hi! I am using the work pool Azure Container Instance:push. For specific deployments which uses this work-pool, how can I override the docker image url?

Hello! I am currently running into issues when using @tasks in a flow, we are iterating through a lot of dates and pass very large pandas dataframes to a task for each date, what I notice is that when you do this after about the 10th day our flow gets OOMKilled, if we don't use tasks it doesn't seem to have an issue. The problem being our upsert method is currently a task. My assumption is that the flow is keeping track of the parameters sent to each task and effectively running out of memory by trying to keep each of our dataframes in memory. Is there a way to avoid this or any information on this kind of thing? 🙂

Hello, Is there a way to make the flow end in a failed state if any logger.error was printed or if any exception was raised?

Is there a good example of how to do subflows on your local account & on a local server? The documentation is unclear about what you would do if you had 1 flow in Python File 1 and 1 in Python File 2 and you wanted to schedule them as subflows together in Python File 3.

Prefect 2.0 migration newbe question ... I created a Organization, with one workspace (added billing and paid with credit card for new users) but when I invite other team members it asks them that there is no organization and asks them to create a space

I am new here and working on understanding migration from Prefect 1 to 2. Is Prefect 1 officially deprecated or just not receiving support from the team anymore?

I'm trying to figure out how to use the prefect cli to get secrets

marvin told me this

but .. he lies..

any thoughts?

Hi 👋 We’ve started seeing some Crash states of our runs.

In the lat couple hours we have started receiving 404s back from api.prefect.cloud when attempting to call flow_runs -- is there something down? it doesnt appear to be related to AWS issues seen today

Creating work pool through CI/CD process: We are using PrefectCloud with Worker Running on ECS Task, and based on the configuration in Deployment it is creating Both flows and infra is written in AWS CDK in the same mono repo and we have plan to deploy this through git hub actions. I understand I can create work-pool through prefect work-pool create CLI however Is it possible to run this through CI/CD where it will only create if the work pool does not exist or update it there is any changes to the work-pool? Currently if I use prefect work-pool name it would then throw error if the work-pool exist. While I could use prefect work-pool ls then read the output programatically and only create if this does not exist, are there any easier way I am missing I should consider when if I want to create work-pool also automatically part of build and deployment? Or possibly I am missing something basic here? Or keen to know if people Automate creation of work pool or that is something one off created through UI, as flow usually changes more compare to work pool on prefect cloud.

hi ! i have been using prefect 2.7.8 version for a very long time . we are trying to upgrade to newer version of prefect to explore newer features. was bit confused by the difference of workers and agents . also when i self hosted prefect server ui (like before) the blocks catalog was empty.. any particular reason for this ? shall i just create them using prefect cli or python sdk , (creating from block catalog was easier )

Bonjorno friends, I have been trying to make the prefect cloud work for the past few days. After resolving various errors and deployment crashes, I encountered a new issue: the agent is raising a NotImplementedError. To address this, I followed the suggestion in an article I found and updated to Python 3.11.5. However, I am now encountering the following error:

(anton) C:\Users\Anton\Desktop\DE2\2_workflow_orchestration\flows\03_deployments>python flow_pandas.py  
Traceback (most recent call last):
  File "C:\Users\Anton\Desktop\DE2\2_workflow_orchestration\flows\03_deployments\flow_pandas.py", line 5, in <module>
    from prefect import flow, task
  File "C:\Users\Anton\anaconda3\envs\anton\Lib\site-packages\prefect\_init_.py", line 37, in <module>
    from prefect.runner import Runner, serve
  File "C:\Users\Anton\anaconda3\envs\anton\Lib\site-packages\prefect\runner.py", line 46, in <module>
    from prefect.client.orchestration import get_client
  File "C:\Users\Anton\anaconda3\envs\anton\Lib\site-packages\prefect\client\orchestration.py", line 11, in <module>
    import pydantic
  File "pydantic\_init_.py", line 2, in init pydantic._init_
  File "pydantic\dataclasses.py", line 41, in init pydantic.dataclasses
    # +=========+=========================================+

ModuleNotFoundError: No module named 'typing_extensions'

I updated all the packages in my env, prefect, python, etc. checked all the possible solutions around 'typing_extensions'. Maybe I missed something obvious? Do you have any suggestions to where should I look? Cheers!

Would Prefect (cloud) be a good choice for a "queue of async logic" type of workload? It's not a typical scheduled data ETL pipeline scenario but instead it's small parts of logic that are triggered by actions of users of your app. For example, in our app, when a user purchases a coupon we want to start a flow that performs some logic (some DB and API calls) in a durable fashion. Specifically, we would use the ECS push work pools. So let's say tens, hundreds of new ECS tasks per minute. I'm wondering if this is a viable use case of Prefect (performance and cost wise), compared to something more traditional like, let's say, a kafka topic with a consumer that performs this logic asynchronously? Thanks!

Hello again! 'm relatively new to working with Prefect, and my current project involves integrating it into my Kedro pipeline. Unfortunately, I've run into some challenges when attempting to run it within a Docker environment. I've been struggling to locate any documentation that specifically addresses the integration of Prefect into a Kedro pipeline project. It seems that implementing Prefect within a Kedro pipeline may require a distinct approach compared to using it in regular Python scripts. I would greatly appreciate any guidance or insights you can provide. Thank you in advance! Also, I'd like to share an additional experience as well. In the past, I faced connectivity problems with Prefect, even though I had started the Prefect agent. I was unable to establish a connection to both the local server and cloud server instances. To resolve this issue, I resorted to starting the entire process from scratch. I'm wondering if there might have been a specific step I missed or if there's something I may have done incorrectly that led to this problem. Your insights on this matter would be greatly appreciated. Thank you once again!

I am working on a Custom Worker and am getting this error in my flow logs

Infrastructure returned without reporting flow run 'ae92eb90-d347-4c4d-b15b-ad4cbf3b5466' as started or raising an error. This behavior is not expected and generally indicates improper implementation of infrastructure. The flow run will not be marked as failed, but an issue may have occurred.

I am returning a 0 status_code and an identifier, so I can't figure out what is triggering this

Good day. I am running into some behavior with the prefect CLI that I do not understand. I have a profiles.toml with local, eks, and cloud entries. Even if I specify

prefect profile use 'eks'

or

prefect profile use 'local'

,

prefect deployment ls

returns the list of deployments from my cloud instance. This also occurs with

prefect -p 'eks' deployment ls

or

prefect -p 'local' deployment ls

.

prefect profile ls

shows all three profiles and shows 'eks' as the active profile. The deploy commands also seems stuck on the cloud profile. Any ideas what's going on?

I was finally able to set up a prefect worker service in my ECS, and I defined an execution role ARN in the task definition. However, I'm still getting the following error when I submit my flow run:

botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support ECR images.

So, in Prefect 1 we would attach schedules to flows with specific arguments (ie, "run with X arguments every day at midnight, run with Y arguments every week"). How do we do this with a Prefect 2 Deployment? Does each one have to have a unique name?

Hello all. Im trying use prefect with my new project, but has some problem with dependency . depends on both pydantic (^2.4.2) and prefect (^2.13.4), version solving failed. prefect use pydantic < 2.0.0. Anyone knows how can i make some workaround about it ?

Hi! I would like to attach some result metadata as key-value pairs to each flow run, so that I could capture some basic metadata like “# of rows copied” or “# of rows in table” and stuff like that. Preferably I would not use results mechanism as having an external storage location feels like an overkill. Is there a way to achieve this?