Hello! I have a use case where we our company has tens of thousands of databases, each database is represented by a client, within a single AWS RDS instance. To have some company wide business analytics performed, we currently backup the instance on a S3 bucket which creates millions of parquet files partitioned by date, database, schema, and table. We currently use an ELT approach using DBT and hence dump all this data into our Data warehouse using a multi tenant design and treat this as our source data for now. Coming to the issue at hand, I've created a flow which consists of multiple sub flows and respective tasks and tested them locally by deploying the workflows on Docker, which are also handled by a Docker hosted Prefect. It all runs well on a subset (100k files) of the data since it is my local machine. But once we deploy our solution to production using ECS Infra and Prefect Cloud, there are some odd issues I'm facing: • I have a flow which scans the metadata of all the millions of files and groups them based on which database and table they belong to. Initially I just simply returned the dictionary from the task and pass it onto the next subflows to start working on them. This worked well on my local machine, but the flows get stuck in PENDING state when running on the same subset (100k files) of data on Prefect Cloud. I was able to narrow it down to one of the parameter which was that large mapping dictionary. As soon as I removed it and tested out the flow, it started running again. • Once I was able to work around this issue, another part of subflow tasks are now stuck in PENDING state due to them taking a subset of that dictionary as input, which would contain a list of thousands of s3 path strings. This also used to work well on my locally deployed workflows, but get stuck when running them on ECS Infra and Prefect Cloud. Just a side note, that all my flows and subflows run on the default ConcurrentTaskRunner setting. It is safe to assume that Flow and Tasks Inputs are most likely serialized and deserialized, which might hint at why these flows or tasks get stuck in Pending states, but I can't seem to understand why it's not being reproduced locally. Is it something related to how Prefect Cloud handles inputs differently compared to Open source variant? Has anyone faced something similar?

feature request: when we run into api request limits that the prefect cloud UI show more than Deployment crashed. Had to dig through pod logs to get the warning as it wasn't getting logged in flow. User Story: As a prefect cloud free tier user I would like clear messaging that my pod crashed not because of Prefect defects but because of API limits. We were almost considering Airflow because we had unexplained deployments crashing all the time and didn't realize they were all due to api limitations not prefect itself.

The "client" for our purposes is the thing that actually runs the Prefect Flow, right? https://twitter.com/dask_dev/status/1648349807340142594

Hoping somebody here is familiar with pdoc3 I am trying to generate documentation for a library with

pdoc3

. It works up until I hit a custom implementation of

prefect.serializers.Serializer

. The error involves the class's

__dispatch_key__

.

What’s the best practice for Data Retention Policy on Prefect deployment runs? Just as a reference, here is how it is implemented for Apache Airflow, as yet another garbage collector DAG: https://stackoverflow.com/questions/66580751/configure-logging-retention-policy-for-apache-airflow I’m sure that Prefect has either a built-in mechanism for that, or encourages a common idiom for rotating / archiving / deleting artifacts from old runs. Context: We have a persistent storage on Azure Blob Storage (the S3 equivalent) where we store artifacts (e.g. output files and images) from a Machine Learning (Kedro) run. The space can pile up pretty quickly across runs and we would run out of storage, rendering our Prefect deployments not operational. What kind of policies are recommended to evict data from old runs? I don’t want to run out of space and I want the Prefect pipelines to remain operational. I know that some of you would say: “_It depends_”, so for the sake of this example let’s imagine that I have a dedicated 256GB of storage. Should I set a threshold (e.g. 70% of full) that will be as a trigger for evicting (removing) artifacts from old runs? Also, when should this run? as the first (prerequisite) subflow in my bigger flow, or as yet another deployment in Prefect on a recurring schedule? Thanks!

Hi @Prefect, We are deploying Prefect Agent to AKS using Helm Charts. In the values.yaml file we are providing the work-pool name. But looks like helm chart deployment runs successfully but it does not create the work-pool mentioned in values.yaml file. For reference purpose see attached copy values.yaml file(removed sensitive information).

values.yaml.yaml

Hi, we are using prefect 2.8.4 and planning to upgrade. I took the latest helm files and updated the same values for postgres which is externally hosted. in existingSecret field, we have 2 values -> connection-string and password and our password starts with numeric and not lowercase alphabet. It works fine with the current version but upgrading it to latest version gives this error

a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. '<http://example.com|example.com>', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'

Is there something that can be done without updating the password as the same database is being used at multiple places and changing the password would be a lot

Hi! I have an agent running on Azure Container Instances. It is randomly stopping after flow completion with this error:

08:17:45.635 | INFO    | prefect.infrastructure.process - Process 'prudent-worm' exited cleanly.

Failed the last 3 attempts.  Please check your environment and configuration.
Examples of recent errors:

Full stack trace in comment

Hi. We're on Prefect 2.8.5 using python 3.11. We're seeing 502 errors. These errors leave the flow in a running state, so no alarms are raised. Any thoughts on how to resolve this?

Hi. We are in the process of upgrading to prefect 2 cloud but in the short term we have a problem with our prefect 1 cloud service. We need more than 10 schedules for a specific flow and these are getting truncated. I can see that in config you can increase

flows.schedule_flow_runs

which would solve this. Could we be provided a method of setting this or for the configuration to be made to our workspace? Thank you.

Hello, I'm trying to run my worker inside a kubernetes pod to avoid restarting it manually everytime it crashes (every few hours so far) I followed this tutorials for agents and adapted it for worker, it launches correctly but the pod does not restart when the worker is unhealthy. Source problem seems that the prefect worker process does not exit when the worker is unhealthy, so kubernetes does not see it's failing. I tried to configure the livenessprobe based on a basic grep (didn't find a way for this because the process still runs when worker is unhealthy), then based on a API request (Marvin suggestions seems to be confused between agents and workers?) but so far I can't make the API requests work (404 on every request i try), and at this point I'm thinking there must be something easier?

Hi All, I have a deployment here to the same work queue, and we tried executing four runs at the same time, but they keep ending up being late. I checked the work agent and it appears that it only works after I press enter in the work agent window. It's not the first time we've seen the work agent stuck, is there any remedy for this? We also notice the agent being 'stuck' when we press retry. Thanks

all links https://docs.prefect.io/latest/concepts/deployments/#examples are not found fyi

I'm getting FileNotFound errors when running my deployments now, using azure SB and docker container IB on prefect 2.8.7 - can anyone help?

Hi @Prefect community, We are facing the error in deleting the work queue using the api: https://api.prefect.cloud/api/accounts/{account_id}/workspaces/{workspace_id}/work_pools/{work_pool_name}/queues/{name}. we are getting either 200 or 500 as code. Please look at this. When we got the 200, work queue has been deleted and added again. Not sure what is going on behind the scenes. Please consider this.

Hey all - I'm getting a 500 response when my prefect agent tries to hit the

<https://api.prefect.cloud/api/accounts/><>/workspace[…]2d2/task_runs/<>/set_state

endpoint. It kicks off the flow just fine (airbyte connection) but quickly runs into this error:

self._records_synced = job_info["attempts"][-1]["attempt"].get(
IndexError: list index out of range

Any ideas? Version info: • Running on an EC2 (also get the error locally), hitting Airbyte version 0.36.2-alpha

>prefect version
Version:             2.8.7
API version:         0.8.4
Python version:      3.8.16

Hello, sometimes we have flows that stays indefinitely in a PENDING state, and we don't have any logs in the agent or anywhere explaining what is going wrong. We're running our flows using the Kubernetes infrastructure. Any idea why they would stay in that state forever without anything else happening ?

Anyone else seeing the flow graph not showing up on chrome? It shows up on safari though.

Hi all, Prefect 1.0 had DatesClock as a schedule type. Can anybody tell me what will be the DatesClock alternative for prefect 2.0?

Hey Prefect Community, I'm having a issue with running a deployment using a docker container. When I run the flow I deployed into a S3 bucket (other deployments using this bucket are working), I get the following error:

Flow could not be retrieved from deployment. Traceback (most recent call last): File "<frozen importlib._bootstrap_external>", line 879, in exec_module File "<frozen importlib._bootstrap_external>", line 1016, in get_code File "<frozen importlib._bootstrap_external>", line 1073, in get_data FileNotFoundError: [Errno 2] No such file or directory: '/opt/flow-code.py' The above exception was the direct cause of the following exception: ScriptError: Script at 'flow-code.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')

Has anyone seen this before? Any tips on where to spend time troubleshooting?

Hi @Prefect, Trying to run a flow but it goes into "Late" state right after it kicks off and then work queue goes into unhealthy state. Any suggestion how-to troubleshoot this issue?

Hi Prefect, is there a way to do retries from the point of failure? ie if the flow kicks off taskA and taskB runs, but taskB fails, how do we retry taskB only?

Hi @Prefect! I have observed two similar issues with our self-hosted Prefect Server and am hoping to get some feedback from the community. Prefect Server setup: We are running Prefect Server v2.9.0 in Google Kubernetes Engine (limits of 8cpu and 16gb memory, excessive to eliminate resources as a cause of the issues), and have a Postgres DB GCP Cloud SQL instance that stores the state (4cpu and 16GB memory, 10GB solid state storage). All Prefect Deployments are configured as Kubernetes Jobs. In general, this setup works as expected. We can run many jobs at the same time, observe their progress, schedule them, cancel them, re-run them, etc. However, we have experienced two issues in the last two weeks that have brought down our Prefect Server (running jobs fail, UI becomes unresponsive). Issues: 1. One of our deployments/jobs creates many many logs (let's say tens of thousands). When that job runs, the logs in the UI become very slow to load. After a couple refreshes, Prefect Server resource usage (cpu and memory) spike, the Postgres metrics spike (cpu usage, ingress/egress bytes, etc), the UI becomes unresponsive, and currently running jobs fail because they loose connection to Prefect Server. Resolution: I can resolve this issue by not looking at the logs for that job. Or, of course, I can just not write so many logs. 2. Another deployment/job returns a large data set from one flow which is consumed by the next flow (let's say around 1GB in size). When this stage of the job runs, the UI becomes unresponsive, Connection Timeout errors are thrown by Prefect Server, Prefect Server cpu and memory spike, Postgres metrics spike, and ultimately Prefect Server goes down. Resolution: The only way I have been able to resolve this issue is to restore the Postgres DB to a previous state. Once restored, if we remove the passing of data from one flow to the next, the issue is resolved. Although both of these issues are avoidable (don't create so many logs, don't pass large dataset between flows), this seems to point to an application issue with how Prefect Server communicates the the Postgres DB. Has anyone else observed similar behavior? Or have suggestions for how to be more resilient to single jobs taking down the entire Prefect Server?

is it possible to track how many tasks were created, how many have been completed/failed. I know you can check in the Task Runs tab but looking for something that gives count. edited: think what am looking for is a progress bar, think v1 had something like that where you could click the mapped task n it will show stats

Hi, I just opened a feature requests https://github.com/PrefectHQ/prefect/issues/9273, what do you think about it ? 🙂

Hey everyone, I'm following this tutorial, but I'm unable to run

prefect project init

I'm facing this error:

No such command 'project'

We noticed that our failed prefect jobs that use kubernetes still keep running, is there a configuration in the prefect deployment that we have missed?

Hey folks, I was wondering if it's possible to make our tasks and flows functions as methods of a class, without having the linter screaming at me (and without disabling type checking). Example of what I mean, try this:

class MyClass:
    @task
    def doStuffWithArgs(self, myArg: str):
        ...

    def doStuffWithoutArgs(self):
        ...

    @flow
    def doMoreStuff(self):
        self.doStuffWithArgs("stuff")
        self.doStuffWithoutArgs()

You will see that for

doStuffWithArgs

Pylance complains that no overload matches that call (see screenshot):

Hi folks, I have a kubernetes infrastucture setup with namespace set to

pefect2-worker-dev

[image below]. When I try to execute a flow run in that infrastructure, it gives out the following error.

Submission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'e6<<SOME STUFF HERE>>37:14 GMT', 'Content-Length': '330'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:prefect2-worker-dev:prefect-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}

I am unsure as to what actually determines which namespace the job is created in. [prefect v2.10.4 on python 3.10]