Hey, I recently ran into this issue:

Failed to load and execute flow run: RuntimeError("can't start new thread")

Restarting the VM resolved the issue but no flows were able to run at all until it was restarted, all with the same or similar errors. Is there any way to fix this without restarting the VM?

Hey, sometimes we miss some CloudTaskRunner Errors in the output of

prefect agent docker start --show-flow-logs

. We can see them in the GUI, but not in the terminal output. Is this a known behavior?

do you have a code example of setting a filter in a certain timezone? i think they default to UTC but i’d like to set them in PST (since the start time is in PST)

Hello, I'm starting to test the Kubernetes Agent on Prefect 1.x....our requirement is to run up to 1000 concurrent jobs. With the standard executor I'm seeing a memory requirement of about 200MB per job, so we'd struggle to reach 1000 concurrent jobs on our moderately sized K8s cluster. Would the Static Dask Cluster approach be a better option? Are there any trade-offs in terms of reliability compared to using Kubernetes jobs - our workload is expected to be a mixture of short running (a minute) and long running (a few minutes to an hour) jobs - which might influence our approach towards reliability.

Hey, I am trying to send a notification with the state of a task when it finishes/fails. Something like "X completed with status: success". I didn't really find anything similar here other than the get_task_run_result but that seemed overkill? Also not sure how that would work to get the status? Is there any way I can call the state of the task like below?

run = DbtShellTask()

    send_notification = send_message(title = "Run Completed" , body = "The run has completed with status: "  + run.status)

At the same time, it would also be great to return any error messages on a failed run. Any idea how to do that?

Hi - we are running into a problem with running a flow on a docker container. We get the following error:

Failed to load and execute flow run: ModuleNotFoundError("No module named '/root/'")

Our setup: • a GitHub agent packages and deploys a docker container to ECR • ECS pulls the ECR image and deploys a fargate instance • (For our proof of concept) When the machine runs, we register a flow that lives in the image/fargate instance • After that we run

prefect local agent start

(both flow registration and starting the agent are run from a script that our ECS task definition runs on boot) • We do a “quick run” and get the module not found error. Other things to note: • all of our flows and supporting modules are on this container. • I’ve read some of the existing discussion surrounding this error and was not sure if the solution applied to our situation - given that we are running everything on the same Fargate instance

Is it possible to update a registered flow's project value (effectively moving it to a different project without changing anything else, like the flow's run history, etc.)?

Hi, Yesterday we faced 2 issues with Prefect flow which uses K8s job_template. When job started it started to fail with not very useful:

'str' object has no attribute 'setdefault'

after examining code we understood that issue comes from https://github.com/PrefectHQ/prefect/blob/9b5fb783d64299a180e5a54ed88ca6d4a1f8fb90/src/prefect/agent/kubernetes/agent.py#L490 . It could be an access issue or missing file, or even a typo in yaml file. There is no error handling or checks to point to root cause properly. MIght be a good minor story to add checks and throw proper exception.

Hi. We are using prefect 0.15.13 server version. The graphql (v2.1.1) throws the below error. It persists even after multiple restarts.

Hi, I have been asking for help some time ago, regarding a issue of tasks and flows getting longer and longer to complete until the whole application freeze (my processors being filled with many processes). I have been told to dockerize the app through the guide provided in this link, which I have done, also because it made our deployment process cleaner and more robust. That was quite some work but it has been making our job a lot easier. Unfortunately, we can still observe the same pattern today, though less extreme. I have joined a screenshot of a flow. It almost always fails because it maps a task to fetch data from different sources, and one of them is down at the moment. The flow takes longer to complete not because of the volume of data to fetch. Indeed, it gathers data on a daily basis: every half-hour, it fetches all the data, accumulating throughout the day, for the current date, and it takes longer today at 1am (without much data then) than yesterday at 11pm. When i redeploy the application, the flows gets back to the normal in terms of speed. I've done that this morning, which is why the bars are shorter at the end of the screenshot. I am kinda lost, as we risk, without close monitoring, a failure of our whole environment. The issue makes me think of the Airflow's instable scheduler one. As a matter of fact, the fix I've thought of is to redeploy the application nightly, but I would by far better understand the root cause of the issue. If you have any idea, I'd be delighted to make some further research with you, and maybe we could help Prefect become an even better product. Alexis

Has anyone trying to build/send a flow(docker storage) from github-runner? Basically, this is what I’m doing: • prefect up and running on k8s • trigger a build by pushing a flow to github on specific branch • a self hosted github-runner on k8s that is waiting to exec a job ◦ github runner has docker in docker enabled and can communicate with prefect-server-apollo api ◦ prefect is installed on my github-runner ◦

~/.prefect/config.toml

looks like this:

backend = "server"
[server]
endpoint = "<http://prefect-server-apollo.prefect-server:4200>"

• in

.github/workflows/build.yml

, execute

python flow.py

which it should build as docker storage when ever my github-runner exec

python flow.py

, it won’t build the docker image for me here is the error message:

<http://localhost:2376/version>: Bad Request ("b'Client sent an HTTP request to an HTTPS server.'")

Any thoughts/suggestions would be much appreciate it!

is there a way to set dependencies between two flows (not tasks)?

I'm running prefect server on a EC2 machine using

--expose

and connecting to the UI using the server IP address on port

8080

. All that works fine, however I noticed that the url in Slack notifications are using

<http://localhost:4200>

. How can I configure this part?

Hello, is it possible to use multiple different images for tasks in a prefect flow? And with a Kubernetes runner or Dask runner?

Hello community, I have prefect prefect in a docker over a linux VM. is it really necesary or better to move to prefect 2.0 I already have around 20 scripts that are loaded in v0.15.3. If so what is the best way to upgrade to v2 or v1?

When is Prefect 2.0 Cloud going to be GA ?

Does Prefect Cloud have any certifications FedRAMP or otherwise ?

Hello! I'm running a Prefect Server on K8s and instantiating a Dask cluster to run my flows. It seems to be working well (the flow finishes successfully), but it is not storing the task logs on the UI. Is there any way to guarantee it is going to be there? When I've added a

sleep(60)

before the end of the task, the task logs appeared, but without it it does not work.

Hi, on prefect 1.x could you confirm when task run state is updated in the core prefect database. I'd like to understand if we have an agent failure/restart what would happen to "in-flight" workflows - would the agent be able to process the workflow from where it left off (re-running the last task run)

few Questions about Prefect 2.0 [Orion]. 1. Why is there no Helm chart for Prefect 2.0 Orion 2. How many components are then in Prefect 2.0. Orion 3. Where can I find the components and Architecture of Prefect 2.0 Orion

I have a self-hosted prefect server running 0.15.13 that I am trying to set up a Slack integration for via webhook. I am clicking on the "secret link" in the docs to get the webhook URL. But when I paste this into the Prefect UI and click Save, I get the error:

Variable "$input" got invalid value null at "input.tenant_id"; Expected non-nullable type UUID! not to be null.

I searched the community and saw some recent discussion about

tenant_id

in this forum related to the 1.0 release, but none in this exact context I think. Do I need to install a newer version of Prefect, or click an older version of the secret link, to make this work?

hi all - i am getting a pickling error on something related to SQLAlchemy when i go to register my flows to s3 (AWS prefect cloud setup). I recently tweaked some db connections (snowflake) to make them more performant; but can't find anything in SO related to this. has anyone ever seen something like this before?

Seeing a new pattern today running tasks on ECS. The prefect logs show

Flow run SUCCESS: all reference tasks succeeded

and then the next log line says

No heartbeat detected from the remote task; marking the run as failed.

Hi, How to register,

flow.register(...)

, a docker agent ? Can I have different docker agents running on a single server?

Hi, my flow of flows is returning a flow like the image attached. Is there a way to get the names of what it’s doing? e.g. sub flow’s name or task names?

I have setup docker and Prefect on remote on-premise server. How should I keep the Prefect server with docker agent always running?

Getting these errors & warning while starting the server Env: python 3.6.9

What’s the best way to prune\cleanup old data on Prefect Server? the

.prefect

directory reached a size of 428G.

EDIT It seems I was seeing something else, please disregard :) Hi! I’m not sure if this has been discussed before… I’m running Prefect Server on EC2 along with an ECS agent deployed as a fargate task. I’m seeing the issue that if the server ever goes down (or the connection is lost otherwise), the agent will show a

requests.exceptions.ConnectionError

, but does not exit. However it’s also does not seem to reestablish the connection once the server becomes available again. Is there some way to either 1. have the agent retry connecting some time after encountering a connection error, or 2. exit after a connection error so ECS can detect that and restart it? The way it looks now I have to manually restart the agent in this situation, which I don’t think is ideal… Am I missing something? Thanks in advance!

Hi all, I have having an issue with our logging implementation changes are not holding when registered to and running in Prefect Cloud. Essentially, the custom log handler (which is just opentelemetry-sdk) enforces a new

_log_record_context_injector

function to modify the names of

PREFECT_LOG_RECORD_ATTRIBUTES

for our purposes which is then added to the root Prefect logger. This function call occurs in a Task class I created which is set as an

upstream_task.

When this is run locally, all log records have the corrected attribute names. When run in Cloud, only the task run to initialize the log handler enforces the changes. Given that all tasks should inherent from the root logger, why are the changes not being implemented across the tasks/flow? Further info: we are using KubernetesRun with Docker flow storage. Also the changes hold when the implementation is run locally.