Any chance the API had some issues last night? It seems like the flow shut down before it started because it couldn't pull secrets. No errors before or after this one bad run:

Failed to load and execute flow run: ClientError([{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])

Heya! How do we add tags to AWS ECS containers launched by Prefect? Like, what arguments do we pass to

DaskExecutor

and/or

ECSRun

?

Hi Community, I’m using Prefect 1.0, and I want to achieve a parallel running using Dask. Here is the process: We have a flow of flows framework as below, each A, B C, D E is a flow to schedule some transformation jobs. For flow C, D, E, we wanna pass some org_id as a list of variables to the flow (which scheduled dbt jobs), and with parallelism for example, org_1, org_2 and org_3 can run in parallel and accelerate our ETL process. Question: 1. how and where can I set the list of org_id, since in the future we wanna add more org_id, so it cannot be hard coded, must have some certain flexibilities 2. to achieve the parallel running, where do I set DaskExecutor? in single flow or in the flow orchestrator?

Are there any docs on how to use the Postgres Task while connecting via an ssh tunnel ? AKA A bastion server

P2.0, running

prefect work-queue ls

I get below:

Response: {'detail': 'Request specified API version 0.6.0 but this server only supports version 0.7.0 and below.'}

For more information check: <https://httpstatuses.com/400>

An exception occurred.

When using Prefect 1 or 0.15, how can I know for a task with

always_run

trigger and

skip_on_upstream_skip=False

, whether any of the upstream-tasks failed or were skipped? I need this information in the task-run method.

Team, Can someone share an example (Prefect 2.0) to have a flow with kwargs and pass parameter in the DeploymentSpec for the flow? So that I can pass many parameters dynamically. Thank you for your help.

Prefect 1.0 - Why Prefect don’t recommend forcing version update when the flow metadata didn’t change? Having an updated timestamp on the flow helps user understand when the flow was updated without going back to git repos or CI/CD pipelines logs.

Hi all! 👋 As we get close to Prefect 2.0 GA next week the Prefect team is making lots of releases. Yesterday’s 2.0b10 release fixed some minor bugs. 2.0b11 was released about an hour ago. We don’t have formal release notes, but highlights include the following changes: • Infrastructure types are added and preferred instead of flow runners • Default storage and storage blocks are removed • Results are always persisted locally instead Be prepared for more 2.0 changes in the coming days. We’ll be updating docs and examples in Discourse as quickly as possible. Don’t hesitate to reach out here with questions. 🙂 Finally, a reminder that if you use 1.0 and don’t want to install prefect 2.0 when GA lands next week, make sure you’ve pinned your version (for example:

pip install prefect==1.2.4

.

Hi All, We are running prefect in ECS cluster where the flows run as fargate tasks and its working just fine.. The only problem we are seeing is overall time taken by a flow to process, which again is not something prefect is causing, but the fargate task itself is taking around 60 sec to just startup and be ready to execute the code. We want to switch to a better way. Idea is to have flow also call an 'always running' container which will be a EC2 based ECS service with 1 or more tasks, and the flow should access those containers only, rather than creating a fargate task for each flow. Is this possible and supported by prefect ? If yes, could you please direct me to the correct documentation ?

Hi All, I’m using https://cloud.prefect.io/, meet some issues, pls help add ‘1’ in sys path as ‘2’ use some libs under ‘1’ this works fine when I use flow.run(); but when I use flow.register(), the agent(already restart) meet this: It looks like the agent not understand sys.path added.

S3StorageBlock is not working with Minio as we cannot specify the endpoint_url. This is something I added on the collection-aws. Will this block move to the collection-aws ? If not, how can I do a push request to the branch ?

Hi 👋 I’m testing Prefect 2.0 kubernetes/Helm deployment Everything seems fine expected that the agent got an issue when launching the job because the namespace and the service account are set with default value. Is it possible to override these value ? I didn’t found it in the doc

Hey. I just upgraded to prefect 2.0b11 and I've an agent running in a rather restrictive kubernetes environment - meaning I can't run it as root. This results in having this warning at the Agent startup:

/usr/local/lib/python3.9/site-packages/prefect/context.py:461: UserWarning: Failed to create the Prefect home directory at /.prefect
with SettingsContext(profile=profile, settings=new_settings) as ctx:

While Agent still starts up and picks up flow-runs, I get the following exception during flow-run

File "/usr/local/lib/python3.9/pathlib.py", line 1323, in mkdir
self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/.prefect/storage'

I guess this has something to do with removing storage/default storage in the latest release? I configured my deployment with a RemoteFileSystem - Packager - not sure why prefect still needs to write something locally? Find my flow + deployment in the Thread.

Does prefect enterprise support onpremise deployment?

Hey, On the last Discourse announcement it says that "*If you currently use Prefect 1.0, please make sure you’ve pinned your Prefect version explicitly* (including your Docker images!)". What does this mean exactly?

Hi. Prefect Cloud UI should support 1.x and 2.x flows after the Prefect 2.0 GA?

Hi ! I was testing Prefect Cloud 2.0 and tonight, all my block storage for cloud (Azure and AWS S3) disappear from the interface. I cannot create anymore now. Anyone with the same problem ?

Hey, we're using the prefect mapping functionality in Prefect 1. Whereby one of the 5 tasks fails, this however in the UI it is appearing as a successful flow run even though one fails. Is there a neat way to fail the flow if any of the mapped tasks fail?

Hi, I’m using 2.0b9 with orion and the prefect agent running on the same host. My prefect agent is dying frequently (it rarely stays alive for an hour) with exceptions like

anyio.BrokenResourceError

and

httpcore.ReadError

when the agent is using the orion client. Has anyone else had similar issues?

Hey, we are trying to automate sending Prefect invites to users and assigning them a default role. I was just wondering if there's documentation or a query already available that I can test with. We are using Prefect Cloud 1.0.

prefect 2.0 . storage not creating

Are there any limits on what types can be parameters for a flow (or subflow)? I am trying to pass a pandas dataframe from one flow to another, and I get a pydantic error

RuntimeError: no validator found for <class 'pandas.core.frame.DataFrame'>

, so I tried to use

validate_parameters=False

in the flow annotation, but it made no difference. What am I doing wrong?

Hi, I have been experiencing some issues with Prefect Docker Agents in our deployment: • we setup agents manually with

prefect docker agent

on a single server; • then we run several flows; • we make updates to flows, and each update creates a new version of docker image; with each run the docker images are pulled onto the machine. After some point we run out of disk space

Has anyone encountered and solved a similar issue before?

Hello, I've noticed tasks start up, run and succeed. Then start up again a few minutes later. Is this expected behavior? Is there a way to ensure this doesn't happen?

Hello, I am trying to schedule metadata ingestion flows for Amundsen using Prefect. The Amundsen code that I am trying to schedule has run() function. So on trying to run the code as part of task is failing. Do you have any suggestions to schedule such type of flows? We are currently using Perfect1.0 Thank you, Divya

Thanks so much to @Jeshua Irving and @Mathijs Miermans for chatting about their experience with Prefect today. The recording of our fireside chat is on our YouTube channel.

do you absolutely have to explicitly specify an ECS cluster name to an

ECSRun

config or can it not infer the correct cluster from providing agent labels?

Hi All, I’m running into an issue with the 2.0b12 version.

root@c0713100a1dd:/opt/app# python
Python 3.8.13 (default, Jul 13 2022, 05:54:24)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import prefect
>>> import prefect.flow_runners
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'prefect.flow_runners'
>>> prefect.__version__
'2.0b12'
>>>

It looks like we’ve removed the flow_runners module, but it doesn’t appear reflected in the documentation. Version was published earlier today