Hi Kelv, Jobs should be transient - can you give some examples of the names of the pods you are removing ?

You can also pause the work queue / deployment

Hi all - I’ve been able to successfully deploy a flow to prefect 2.0 using a kubernetes-job block and custom image that points my internal docker registry - however when I try to run the flow, I’m getting a k8s error from the agent:

stern --tail 500 prefect-orion

"message":"The POST operation against Job.batch could not be completed at this time, please try again.","reason":"ServerTimeout"

Hello Frank Hardisty of Replica here. We had our Prefect logs turned off due to extreme log spam. Brett Naul and I implemented a fix that should prevent that source of log spam from happening again, could we please have Prefect logs turned back on?

Hi, everyone! We are using prefect 1.2.4 and recently got the following problem when started a flow from CLI with

-w

option:

f"`watch_flow_run` timed out after "
RuntimeError: `watch_flow_run` timed out after 12.0 hours of waiting for completion. Your flow run is still in state: <Running: "Running flow.">

Is there a way to pass a different max_duration to a

watch_flow_run

in such usecase?

Hi everyone, I was wondering why when creating subflow and passing parameters in UI schema prefect creates additional List and Dict tasks. It adds unnecessary complexity and visual noise in our schema, maybe it‘s possible somehow create subflows without these additional downstream Dict and List tasks in UI? :)

Hi everyone, we are using Prefect to run a job that connects to redshift cluster and unloads files to s3 bucket. It's been working fine in the dev environment but when we try in test environment, we get this error. I was wondering if anyone has an idea on what the issue could be. Thanks!

Just a shoutout to the prefect team for v2.4.0. 🎉 😛arty-parrot: This release actually solves my two last annoyances I had with prefect 2.0, namely not being able to set custom parameters and schedule timezones via CLI. • Support for setting custom parameter values directly from the deployment build CLI • New CLI for interacting with and setting custom timezone-dependent schedules for your deployments incl. the ability to pause or resume a schedule directly from the CLI Thanks to the team for this awesome release and it's incredible to see how fast prefect 2.0 moves in the perfect direction.

Hi Prefect Community! Do you know if it is already possible to specify cpu/gpu resource requirements per task for the RayTaskRunner? So far I only managed to configure the Ray Cluster, but I am not sure if its possible to map the tasks to the different worker types to then autoscale accordingly (for example only scaling out gpu workers if needed).

Hey all, I am trying to run flows using Docker as infrastructure. I’ve set the pull policy to never because I’m currently performing tests and the image is not public — in other words, I am forcing the agent to use the local ones. So far, so good. The issue arises when running the flow, the container will try to run

python -m prefect.engine flow_uuid

, it fetches the flow but then tries to copy files into the same folder, in this case

deployment.path

is the

cwd

&

pwd

which in turn means that

.

resolves to

deployment.path

— making the

filesystems.LocalFileSystem.get_directory

copy the files into themselves (through

shutil.copytree

) which ends up in a HUGE error like so:

[<redacted for brevity>, '.', 'c', 'f', 'g', "'", '>', ' ', 'a', 'n', 'd', ' ', "'", '.', '/', '.', 'i', 's', 'o', 'r', 't', '.', 'c', 'f', 'g', "'", ' ', 'a', 'r', 'e', ' ', 't', 'h', 'e', ' ', 's', 'a', 'm', 'e', ' ', 'f', 'i', 'l', 'e', '<', 'D', 'i', 'r', 'E', 'n', 't', 'r', 'y', ' ', "'", '.', 'f', 'l', 'a', 'k', 'e', '8', "'", '>', ' ', 'a', 'n', 'd', ' ', "'", '.', '/', '.', 'f', 'l', 'a', 'k', 'e', '8', "'", ' ', 'a', 'r', 'e', ' ', 't', 'h', 'e', ' ', 's', 'a', 'm', 'e', ' ', 'f', 'i', 'l', 'e']

With regards to Prefect’s code, I have a solution, just check if both paths match and don’t copy in that case, but I’d like to get some feedback from the Prefect team before reporting a bug and sending in a fix — i.e. if I am doing something wrong.

Hello community, Anyone experienced on setting up ECS Agents? I successfully register the flow and the agent, the task appears in AWS on the designated Cluster but nothing happens (the Prefect Task remains pending) Please, leave here any useful resource 😄 thanks

In Prefect 2, what is the equivalent to the Module storage from Prefect 1? https://docs-v1.prefect.io/api/latest/storage.html#module

Hi all, I am running into issues with my agent being able to submit flows to my infrastructure: Prefect Version: 2.3.0 Infrastructure: KubernetesJob Traceback:

| ERROR   | prefect.agent - Failed to submit flow run 'FLOW_RUN_ID' to infrastructure.

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"The POST operation against Job.batch could not be completed at this time, please try again.","reason":"ServerTimeout","details":{"name":"POST","group":"batch","kind":"Job"},"code":500}

More configurations in thread

Is there a Prefect/python-native way for a flow to trigger different deployments? I have a few deployments that have different Docker and k8's requirements, but are related. A simple example is one deployment loading data to s3 and another deployment running dbt on that data. It would be incredible if I could create a parent flow which triggered the pre-defined deployments sequentially. Some (hacky-ish) solutions I can think of: • Define the logic in one big flow and consolidate to one Docker container • Create a flow with the prefect-shell collection and run deployment CLI commands to trigger the deployments sequential These solutions would definitely work as expected. I am curious if there is another solution using the new python

Deployment

class (not seeing anything in the API reference). Or, if anyone else would find a

.run()

method on the

Deployment

class useful? 😼

HI all, I'm new to prefect, loving it so far. I'm working on a 1.0 project that I've inherited - I was wondering if there is a way to utilize the graphql api to verify that all downstream tasks (including children of children etc) of a particular task run were successful. I can do it via repeated querying for downstream_task info, but it would be nice if there was some other method that didn't involve hitting the api repeatedly. Thanks.

https://medium.com/the-prefect-blog/serverless-real-time-data-pipelines-on-aws-with-prefect-ecs-and-github-actions-1737c80da3f5 I'm trying to use the while true to run a flow. It works locally and keeps on creating new flows like I want. When I run this through the cloud and a helm prefect-agent on GKE, it runs one flow then stops.

Hey all, we’re hosting @Oscar Björhn, a Club42 member, on PrefectLive today at 3p Eastern where he’s going to show us how he integrates Prefect into his workflow, including using different flows with different environments inside a Docker container. Come join us live on Twitch!

Hi there. I have a flow that is running on ECS (Prefect 1.0). I see that some tasks got stuck in running state and they have been there for a long time. How can we avoid this? Can prefect cloud do something about them? Is it possible to set a timeout at the task level? I manually stopped them but it would be good to have an automated way of killing these if they run for more than X hours. Thanks!

Hello - have a quick question regarding the local dask executor: We have stumbled upon this issue as well and we are working with prefect version 0.13.10. So it seems that the examples as suggested in prefect docs for LocalDaskExecutor dont work and we need to go via this RemoteEnvironment variable. Has this been addressed at later versions of prefect or should we just proceed with the RemoteEnvironment solution ?

It looks like create_flow_run is internally looking for a Cloud URL. Is there a corresponding call for Prefect Core that doesn't need a cloud URL?

Hi, I'm curious if there are project count limits on Prefect Cloud?

I keep getting this error when trying to have a scheduled run from a deployed flow. It runs fine as a non-deployed flow (when I run it from my terminal). I built and applied the deployment on another flow/script and it worked (but it also doesn’t have any non-prefect imports). Should I

import psycopg2

from somewhere else instead? Running 2.3.2

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/tmp/tmpcre52tf5prefect/SCRIPTNAME.py", line 3, in <module>
    import psycopg2
ModuleNotFoundError: No module named 'psycopg2'

Hi, I'm just starting to use prefect 2.0. I noticed the prefect-email collection. Is it possible to call a new flow from a task?

Hi guys, I need super urgent help!! My prefect UI stopped loading suddenly!

Hello! A flow run was created by a nonexistent user. Is this expected?

I have a flow function decorated with @flow and specify a DaskTaskRunner. I thought this would cause my tasks to get run on my Dask cluster but I saw this in your documentation: “To run your task with a task runner, you must call the task with .submit().” Is this still true even if my flow function is configured as described above?

does an agent that services a work queue with concurrency 3 end up running three python threads (single python interpreter)? if so, does each thread have it's own event loop?

Can flow run context be fetched within a task in 2.0?