Is there a Prefect roadmap available? I'm curious to know what's next :)

Hi is there a way to define a deployment in prefect.yaml that just runs a shell script without defining the flow in python? im using prefect 3.

Hi everyone, I use local storage for result persistence. I'd like to set different paths for different workflows that are deployed together (through the

.to_deployment

+

serve

scheme). If I set the environment variable

PREFECT_RESULTS_LOCAL_STORAGE_PATH

, it's set to the same value for all workflows. I asked Marvin and I was told to set

result_storage

in the flow decorator, either to

LocalFileSystem(basepath=path1)

or

Path(path1)

but it fails. Somehow I need to call

.save()

on the

LocalFileSystem

instance beforehand to create a block server-side. I find this a bit convoluted for a fairly simple need, especially since I otherwise don't need to do it when using the

PREFECT_RESULTS_LOCAL_STORAGE_PATH

variable set to arbitrary values. Marvin also told me that I could pass

PREFECT_RESULTS_LOCAL_STORAGE_PATH

as a job variable to the deployment, but that it was not the recommended solution. What would you recommend? Also, if there are no other solutions than what I described, I would argue that being allowed to simply provide a

Path

object to

result_storage

would be a reasonable behaviour to expect and I would be interested in discussing it or filing an issue on GitHub. Thanks!

Hi All. I am one of the thousands of newly unemployed data analysts/engineers, and I am looking to continue my upskilling to include Prefect, while also finding an online public dataset that is continuously updated that I can do batch, or micro-batch extracts from to populate an iceberg-based data lakehouse, while also applying dbt to the workflow to build various analytics views in PowerBI, Tableau, and Apache Superset. This is all to create a demonstratable portfolio in github or gitlab CI/CD. Before I spend hours looking for that right candidate dataset, could any of you already suggest a public dataset, with a URL? Thanks in advance.

Hi, Prefect Assets is currently only for cloud version, is there any plan to have it on open-source version? That's a very cool feature.

yo do you guys have problems w/ prefect crashing on high task loads? i've been trying to make prefect work for us for a number of months, but getting a bit discouraged: whenever a flow queues up thousands of tasks (on the order of 10,000+) prefect just outright crashes i'm using redis as the message broker, have the background services spun up separately, etc. i've resorted to writing my own very basic task queue, which works --- but given that such a simple (but important) piece of architecture solves the problem, i'm assuming i'm doing something terribly wrong? [basically: "don't queue up more than a couple-hundred tasks on prefect. wait until there are fewer than 100, then queue up 300"] but that's the kind of thing i'd want prefect itself to deal with

it's been kind of a struggle 😞

Hi Everyone, I am using prefect for some automations. .I am using same configurations in both staging and production for deployment .but in staging if I make any change ,the deployment is taking the new code .but in production it is not happening. Anyone knows what will be the issue

Is there any way to disable the Oops. something went wrong pop ups on prefect 3? I’m worried staff against using prefect where I work will use that as a reason for the company to stop using it completely. Seems to mostly be caused by database uniqueness errors in Postgres and the database being locked on development machines.

Is there anywhere I can get the prefect icon for my slack webhook integration?

Hi everyone, does anyone have faced the following? We have a self hosted prefect with EKS and we do use the official prefect helm charts In order to safe resources, and because our flows are really simple... instead of running workers kubernetes types, we switch them to run process types. After few days running this way, we noticed the following: • We have runs indefinetly in RUNNING status. They never finish. It happens because we have autoscaling, and whenever we downscale, the worker pod receives the sigterm, we cal clearly see two log lines about SIGTERM, but it stops immediately. It''s like the worker process receives the sigterm, but it does not wait to anything, just stops, so it does not gracefully stops. The deployment has a terminationGracePeriod defined of 60sec, but it does not matter as the pod automatically shutsdown without waiting. We tried to upgrade to the latest version of the image the helm chart could use ( prefecthq/prefect:3.6.1-python3.11-kubernetes) but we are having problems too cc @Marvin

Hi everyone, we experienced a bug where after re-deploying a flow, the prefect server created duplicate scheduled runs of the flow. when we manually disabled the schedule and re-enabled it, the server correctly only generated single runs. is this a known issue with updating existing deployments?

https://prefect-community.slack.com/archives/C04DZJC94DC/p1763146961194199?thread_ts=1763145223.026799&cid=C04DZJC94DC Is this right that a subflow inherits the

result_storage

from the parent flow, even if the subflow had its own

result_storage

set?

Hi everyone, i'm trying to build a pipeline which have the following components: • Schedule Coordinator: Receive event through a queue and parse the event for scheduling job. We are using github storage to create the flow (to avoid import many dependencies of different workflow into scheduler). • A workpool in K8s • Worker running flow in K8s also Everything is working but we have a problem with the Scheduler Coordinator caused OOM error on K8s even we have tried to increase the memory for the pod, the problem is happened when the schedule coordinator trying to pull github (our github repo size is only 5 MB and we already increase the memory of the pod to 1Gb). Is anyone have this problem before? And beside using github storage to create a flow, is there any different way to create flow without needing to import dependencies of these flows? Thanks!

Hi everyone - my company is evaluating prefect, and have a question about static typing (pyright). It appears to me that if you have a function defined with some input parameters, the

@task

annotation will hide those parameters from the typechecker. e.g., this code will not throw an error:

@task
def example_task(b: bool):
  return False

@flow
def example_flow():
  return example_task()

Is this expected behavior? Did I misconfigure something?

Second question - we use django's ORM for db models. Are there any issues with passing these models between tasks? I think they're serializable.

Hello, I was wondering if it would be better to start the worker from the gitlab ci or using system service?

Hi everyone, we are running prefect3 on k8s and for some automations we have a time delay of 15minutes. Automation is to send a notification if a flow run enters Completed status. Sometimes the automation is immediate but most of the time there is a 15min delay. Any ideas what this could be?

I'm seeing variables change within a mapped task where I have the task call a function. The function prints out the row mapped by the dataframe and returns that value, but later uses of the variable have different values than what was set by the function previously. Did something change from v2 to v3? Should I make the function a task to have it behave normally?

hey, nothing that I'm aware of. Could you share your account ID and more specifically what you're seeing?

Hi everyone, I am trying to run ECS Tasks on AWS. I setup a self-hosted Prefect server, and a worker service that submits tasks to ECS. However, these tasks are not running correctly. These tasks are Docker containers that I have deployed on the self-hosted prefect server (and I run them on there). I think the main problem is that Fargate is running on x86_64 and these are ARM. I can't figure out how to get ARM running on Fargate and if I switch the built to x86_64/AMD, then the container isn't pulled correctly from ECS. I am not sure how to remedy this and would like some help in figuring it out. I have pasted the details in the comment on the yaml/code. I have tried editing the base worker template and nothing happens. I decided to then create a custom task definition and now instead of getting the failure

exec /usr/bin/tini: exec format error

I got the PREFECT in ASCII and then nothing which I imagine is because the Prefect Task/RunId is invalid? I'm not sure what to make of the entire thing and I am at a loss of what to do from here.

Self hosted, v 3.4.25, Kubernetes worker Concurrency limit set to 1 at deployment level, policy set to cancel new Common behavior: duplicate runs start, several seconds apart, one gets cancelled -- Why, has anyone seen this? Example yaml deployment config:

- name: other
  description: XXX
  entrypoint: XXXX:build_other
  schedules:
      - cron: "0 6 * * *"
        timezone: "Europe/Prague"
        active: true
  parameters: {}
  work_pool: *transformations
  pull: *work_dir
  concurrency_limit:
    limit: 1
    collision_strategy: "CANCEL_NEW"

Hello all, I am doing Gitlab migration with my current private repo using Prefect Cloud. However I am struggling with the pull step using the PAT (Personal or Project Access Token). As I understand based on the code in

prefect/runner/storage.py

I have to setup the token as

oauth2:<token>@gitlab.corp-name/group/project.git

However this does not work and i keep getting error 128. Any idea how to resolve this one? Or maybe some workaround that does not depends on pulling perhaps ?

Hello. Is there any way to set up retention for Flow Runs in Prefect OSS? And also something for other tables in Postgres, such as

log

etc. We found e.g. the

PREFECT_EVENTS_RETENTION_PERIOD="2d"

for

events

table. But is there any good "Overall Retention" setup guide?

Hi all - the new UI/UX for Prefect is entirely unusable. It makes zero sense. Is there any way to roll it back?

Hi all, my company has been using prefect using a cloud run work pool for nearly 1 year now. I was investigating push work pools, so I deleted my old cloud run work pool temporarily (since free tier accounts have a max of 1 allowed). Then when I went to create a new work pool I am only able to use prefect’s managed work pool without upgrading?! The managed work pool doesn’t even allow for custom docker images. I was not clear that deleting my old cloud run work pool would regress the amount of functionality I could have on my free tier account! Please could someone advise what my options are here? Besides upgrading to pay 100$ / month?

Has anyone been having issues with pulls_from_block on github? Since yesterdays github outage i’ve been having

Failed to pull from remote:
 Cloning into '/tmp/tmp0h4b39iiprefect'...
fatal: unable to access '<https://github.com/org/repo.git/>': Failed to connect to <http://github.com|github.com> port 443 after 131017 ms: Couldn't connect to server

errors on most of my deployments

Hi all, has anyone been having issues with prefect cloud automation? Usually during midnight, it fails to report/detect the failed deployments/flows. Parent flow and child flow failed at the same timestamp. Parent flows have

prod

tag but child flows not. anyone with the same issue? during daytime its fine. super weird.

Hi, I just signed up for a hobby tier Prefect plan to try things out but can't seem to create even one workspace (which I think is required for me to do anything). What am I missing?

Hi all, why can't I add my github credentials to my prefect block? Every time I try to, it just says None in the github repository block??