Hi, I'm trying too create a Prefect Docker Agent in a container passing the docker.sock to do some DinD Is there a way to mount volumes to the children spawned containers ? I tried to mount a volume when starting with --volume option but doesn't seems to work

Hi all 👋 what happens if I register a flow with Github storage, where the storage points to a ref that's different from the ref I'm registering? In other words: 1. I have a

flow.py

flow with

storage=Github(..., ref='main')

2. I checkout a new branch

my-feature-branch

, and make some edits to the flow 3. I re-register the flow from this new branch, but still leave

ref='main'

...so when the Flow is run, the version registered to the Prefect backend is from

my-feature-branch

, but the agent running the flow will have pulled down the version from the

main

branch? Thanks in advance!

Hi can someone help me understand why these flows are failing trying to access cloud secrets - just had 4 in a row; • https://cloud.prefect.io/dat-eng/flow-run/304e4a56-47c4-43e7-a0ff-56712b35ac61 • https://cloud.prefect.io/dat-eng/flow-run/57204063-5800-402e-b4e2-6cc3d6cefca2

prefect.exceptions.ClientError: [{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

Trying to use DbtShellTask on windows but get the following error:

[2022-06-27 17:16:15+0200] ERROR - prefect.DbtShellTask | /bin/bash: C:UsersUSERAD~1AppDataLocalTempprefect-6rbqzzlx: No such file or directory

This line in prefect.tasks.shell.py generates a strange path:

with tempfile.NamedTemporaryFile(prefix="prefect-") as tmp:

Hi all! I am having this error . Any suggestions? Traceback (most recent call last): File "/home/centos/.local/lib/python3.8/site-packages/prefect/agent/agent.py", line 328, in _submit_deploy_flow_run_jobs flow_runs = self._get_flow_run_metadata(flow_run_ids) File "/home/centos/.local/lib/python3.8/site-packages/prefect/agent/agent.py", line 688, in _get_flow_run_metadata result = self.client.graphql(query) File "/home/centos/.local/lib/python3.8/site-packages/prefect/client/client.py", line 464, in graphql raise ClientError(result["errors"]) prefect.exceptions.ClientError: [{'path': ['flow_run', 0, 'id'], 'message': 'Cannot return null for non-nullable field flow_run.id.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

Hi! In prefect 2.0 I notice that if we pass

future

to another task, the task automatically receives the future result value as a variable which is the expected behavior. However this is not the case when we have a flow with subflows and we pass a prefect state from a flow run (that has a return value) to a flow downwards (where we should manually call `.result().result()`to extract the flows result's value) Is this the expected behavior? If yes this seems a little bit counter intuitive to me, where in the case of passing data from tasks Prefect automatically extracts the result value but not when this result is coming from a flow run

Moving this from another thread since it was unrelated to the original. I registered a new version of a flow this morning that started failing with

Failed to retrieve task state with error: ClientError([{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}])

@Mason Menges Pointed me to https://discourse.prefect.io/t/how-to-fix-the-error-prefect-exceptions-clienterror-message-expected-type-uuid-found-could-not-parse-uuid/832 to assist. I ended up having to register the flow 3 times for it to start working. The first registration was the only time there were changes to the flow, I had to register an additional two times with zero changes to get the error to stop. Is there an issue related to this article or anything that can be done to dig into what exactly is going on?

is there a way to set KV store values in the config or with environment variables so that the KV doesn't need to be called? in other words, how can we do fully local execution if the KV store must call the cloud?

Hello prefect community, I am using "manual_only" trigger for the first time and I noticed that when I resume the flow, the variables computed downstream the manual task are not kept in memory. Is that the correct behaviour or am i doing something wrong?Should i take a look how to persistently cache these previous results (https://docs.prefect.io/core/advanced_tutorials/using-results.html) or there is a simpler solution? Thank you for your help!

Hello! I am attempting to set up DBTShellTask and having a problem with locating the dbt_project.yml file. It is located in the dbt/ directory that is cloned from my github repo. I do not see a variable I can use to change to dbt/. The profiles.yml is created in . so it is found without a problem.

dbt_project.yml file [ERROR not found]

Hi all, I'm trying to map over a task in prefect 0.15.13, using a LocalDaskExecutor where

scheduler="processes"

. The input I'm passing to the mapped task is ordered, ideally so that tasks assigned to the first items in the list are run first, however this doesn't seem to be the case. With my machine that has six cores, on a given task run I can see that the mapped tasks with

map_index

values 0, 1, 3, 13, 14, and 15 are run first, but ideally I would want this first batch of tasks to have map index values of 0, 1, 2, 3, 4 and 5. tldr: is it possible to ensure that when mapping a task, the mapped tasks get run in the same order they appear as in the input array?

Running

prefect auth list-tenants

I get below:

ValueError: badly formed hexadecimal UUID string

Nvm user error.

How can we prevent concurrent runs of the same flow? Given my current architecture it’s not safe to have two runs of the same flow at the same time. I see there are some concurrency strategies with labels but I don’t think that will work here given that we want every flow to have this behavior.

im getting an error

zsh: segmentation fault  python app.py

when im trying to run a flow using

LocalDaskExecutor

using threads can someone please help

Hi! In Prefect 2.0 - Python API orion client. How can we create and provide a flow_filter to read_deployments() ? For example how do I get all deployments that have a specific tag using Python? It would be nice if the API docs had more examples especially related to Python API and client

I'm having a problem in the execution layer of my project. I'm using an ECS Fargate agent to run my tasks. I'm following this method to define task containers. I'm also using the logging options mentioned here but the task only gets submitted by the agent. Nothing happens after that and after a while it gets restarted by Lazarus. Could anyone suggest a way to narrow down the problem or a better practice to follow for configuring flows?

Hi everyone, prefect 1 question here. I deployed an ETL flow and i need to keep track of what happened during each flow run such as "how many record have been ingested, how many failed etc". Ideally this would be done at the end of the flow execution so i looked about flow state handlers in the doc. But i couldn't find if i could use "task" data in those state handlers. Do you have any idea ?

Prefect 2.0 (open-source) Hi! I’m currently trying to set up a simple flow to run dbt in the following way: 1. check

dbt source freshness

a. If I get an error, fail the flow 2. run

dbt build

I’ve figured out a way to fail the flow by scanning the result from PrefectFuture but somehow this doesn’t feel right (simplified code snippet in 🧵). Two questions 🙏: 1. Is there a “better” way to handle this? 2. I’d also like to add some kind of notification when the flow fails (email and/or slack) - I can’t find any integration within prefect itself, so would the recommended route be to send a webhook directly in the flow?

Hi all, how does one specify resource limits for a KubernetesFlowRunner in prefect 2? Previously in prefect 1 you could specify cpu_limit, memory_request, etc... I don't see much documentation on how this work in prefect 2, is there anymore examples of more advanced Kubernetes deployment i can see?

Hello community! I came across this youtube video https://mlops.community/watch/orchestrating-machine-learning-workflows-with-prefect_coZmJ1vR417hh2/ Where can I find the code? @Kevin Kho

In case you need auto clean up of k8s jobs in prefect 2.0 you can add the following

customizations

to the k8sflowrunner cc @Kevin Kho

KubernetesFlowRunner(namespace="prefect", customizations=

[

{ "op": "add", "path": "/spec/ttlSecondsAfterFinished","value": 10}

])

Hey guys i'm randomly getting:

Failed to load and execute Flow's environment: StorageError('An error occurred while unpickling the flow:\n ImportError("cannot import name \'MyService\' from partially initialized module \'myflows.services.myservice\' (most likely due to a circular import) (/app/src/myflows/services/myservices.py)")\nThis may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')

This is happening randomly the same flow with same args can work without any issue. There a way to debug this i'm using S3 storage

I am still unable to create projects that have the same names as projects I deleted last week. I've been speaking with @Kevin Kho about it but he is out of the office so I want to see if anyone else can assist with this. This has put me days behind on development work. What the heck is going on here?

I am not being able to run a Flow. The problem is in the execution layer of my project. I'm using an ECS Fargate agent to run my tasks. I'm following this method to define task containers. I'm also using the logging options mentioned here but the task only gets submitted by the agent. Nothing happens after that and after a while it gets restarted by Lazarus. Could anyone suggest a way to narrow down the problem or a better practice to follow for configuring flows?

Hi! This is causing our jobs to fail 😭 . I see this has been fixed in the master branch. Any idea when the next release would be?

What recommended best practice for registering flows? In bottom of .py script with:

flow.register(project_name='tutorial')

When using script based flow storage, is it okay to leave this line on the bottom? Or will it attempt to re-register the flow on scheduled flow run? Or using CLI:

prefect register --path hello_flow.py --project tutorial --label my-label

Hey how do I add a readme to a flow declaration programmatically?

Is there a list of allowable memory specifications for Kubernetes Run? For ECS I only could enter certain values

Hi team, is Prefect Cloud experiencing any issues right now? I’ve noticed very weird CSS making the UI unreadable and my flows are not starting due to some authentication issue with prefect cloud