yo do you guys have problems w/ prefect crashing on high task loads? i've been trying to make prefect work for us for a number of months, but getting a bit discouraged: whenever a flow queues up thousands of tasks (on the order of 10,000+) prefect just outright crashes i'm using redis as the message broker, have the background services spun up separately, etc. i've resorted to writing my own very basic task queue, which works --- but given that such a simple (but important) piece of architecture solves the problem, i'm assuming i'm doing something terribly wrong? [basically: "don't queue up more than a couple-hundred tasks on prefect. wait until there are fewer than 100, then queue up 300"] but that's the kind of thing i'd want prefect itself to deal with

Depends a bit on what the 10,000 tasks are doing - a task is a "monitorable unit of work" and I personally wouldn't want to monitor at that low level of granularity. I'd maybe break it down into subtrees of work such as parent flows or parent tasks, and use concurrency limits for how many subtrees can fire off at once.

@Janet Carson thanks for this! to paraphrase: group, say, 1000 tasks into a single flow instead of assigning each unit of work a task? in our case, it is nice for each data-processing unit of work to be represented by a task, but i'm willing to like group things up a little more if it'll solve this. do concurrency limits affect how many flows/tasks are queued up @ the same time? or just how many can actually run at the same time?

So, I'm not a Prefect employee, but here's an example of breaking a flow down based on my understanding. Let's say you have USA-Flow which starts Region flows with an arg East, Central, West and let's say, East starts multiple instances of State task with an argument --> NewYork, Massachusetts, etc. And maybe that task starts subtasks for every Customer in that state. And maybe that task starts subtasks for each Order or something. A concurrency limit could be applied at any particular level of the tree. Maybe you want a maximum of 1000 customer tasks or 3 states or 1 region. You just don't want to put it at the very lowest level of the tree if the goal is to stop spawning tasks at some point and wait for the queue to drain a bit.

ooooh ok i think i see what you're getting at: use concurrency to limit the number of running flows, which in turn limits the number of generated-tasks

Yeah, I think tasks once spawned have to exist. They just exist in the "waiting for a concurrency slot" state.

oh yeah. we're doing a lot of cpu-heavy data processing so can't make very good usage of threads; we've gotta use processes one difficulty for us is: each of our workers is a machine w/ around 100 CPUs, and each machine should ideally churn through about 15,000 tasks (1 for each CPU). we group our tasks in batches of 15,000 because each machine needs to download a large file w/ 15000 pre-requisite files (i.e., the grouping is a product of the architecture, not an arbitrary number) --- so there's a natural task-flow grouping

You can decide where sub-tasks and sub-flows make sense, too. The tree doesn't have to be entirely flows.

maybe the thing to do here is instead of 15,000 tasks, split each task into "15000 / number-of-CPUs" it's a little less efficient w/ lower resolution, but will reduce the number of tasks by a bunch

I mean, people have posted here about doing bizillions of subtasks, but I'm not really in that space.

huh interesting. thank you for your responses!! this has been pretty helpful

it seems like queues with many thousands of tasks should just be handled by default by a workflow orchestration system 🫠

people do this today, but many thousands of tasks on what timescale?

prefect just outright crashes

what is crashing? e.g. server container is OOMing? how many server processes / how are resources allocated? people do operate at this scale with open source, but you'll need to tweak configuration based on your workload. if you can be more specific on what's going sideways, i can probably offer some knobs you might want to turn

what goes sideways most of the time (or at least the most obvious thing failing) is database related: prefect overloads the database with connections (this happens even in tasks that don't themselves use any DB connections)---as if there's a connection opened for every task-status. scaling the DB up doesn't seem to make much of a difference. most of my configuration alterations have been w/ parameters like ``PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE`` logfire is a great idea though; this is a good opportunity to use it.