Hi , is there a way to deploy multiple data pipelines(one for daily and other pipeline for weekly etc) , would appreciate the help , thanks in advance

@Marvin How to Clean the prefect salute DB data ?

@Marvin Hello, I am currently trying to use prefect (using helm and values.yaml to setup work-pool and workers). The issue is each flow I try to run crashes due to validation error for KubernetesWorkerJobConfiguration env (value is not a valid dict)

@Marvin How can I use the prefect-sqlalchemy package to fetch data from sql server?

@Marvin what problem is this prefect server

Server error ‘500 Internal Server Error’ for url ‘<http://prefect-server:4200/api/logs/>’
2024-09-04T06:00:16.464149854Z Response: {‘exception_message’: ‘Internal Server Error’}
2024-09-04T06:00:17.254531875Z For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/50006:00:17.252>

Hi all, I’m having trouble diagnosing a GitHub storage problem. I’ve created a trivial testing flow similar to the example script-based workflow for GitHub. I’ve deployed a k8s agent using

prefect k8s agent install

. I’ve uploaded my flow to a private GitHub repo and registered it with Prefect. And I’ve added a Cloud Secret called

GITHUB_ACCESS_TOKEN

that holds a valid GitHub personal access token. When I run my flow the agent’s GitHub storage gives me an

UnknownObjectException(404, 'Not Found')

error. If I change the flow to use a different Cloud Secret key for the PAT, such as

access_token_secret='MYKEY'

, then the agent tells me

ValueError('Local Secret "MYKEY" was not found.')

. How can I introspect the kubernetes agent to verify that the GitHub PAT secret is being loaded from Prefect Cloud correctly?

Has anyone encountered the following error when using the ECS agent? Is there an easy way to increase the number of retries or backoff interval?

An error occurred (ThrottlingException) when calling the RegisterTaskDefinition operation (reached max retries: 2): Rate exceeded

I’m running a prefect flow that moves a large amount of files from s3 to google cloud storage. When there are a relatively small amount of files to transfer, it’s quick. But as the files to transfer grow and especially if file size is large, the prefect process takes up increasingly large amounts of memory. I am using the

S3Download

and

GCSUpload

tasks. My suspicion is that the flow is not releasing the memory of the files being transferred. Is there any way to ensure the file contents are being released from memory?

I'm seeing a lot of warnings about CancellationCleanup loop service taking a long time in my prefect server logs:

20:03:50.981 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 30.438319 seconds to run, which is longer than its loop interval of 20.0 seconds.
20:04:24.279 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 33.296899 seconds to run, which is longer than its loop interval of 20.0 seconds.

I've already deleted all the

Cancelled

runs in the hopes of fixing this, but it seems to have no effect. Also, my server's CPU usage is very high, and I'm wondering if it's potentially related to this. Does anyone know what makes CancellationCleanup take so long?

Hey Folks 👋 Lately, some of my flow runs have a "Run Count" of 2 (instead of 1). When looking at the "Results" tab I see that both runs were started pretty much at the same time. We're using k8s infra. What could be the conditions of a single flow run being started more than once? Having a flow launched twice can be quite problematic, especially when they have the same id.