Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi all. I'm new to Prefect (and development). I had a doubt.
I understand that one can use prefect to set up flows that run tasks in a particular order among other things. Does it also implement queueing?
So far in my projects I've only used redis-celery to run big tasks in a queue.
PS, I've mostly worked in web dev.

Hi <@U014XPH5HL1>, welcome! Prefect is not a queueing framework, in that it doesn’t offer you fine-grained control over its queueing mechanisms. It attempts to run work as close to its scheduled time as possible. However, work is queued in the sense that if you don’t have workers or resources available to handle a scheduled job, it will not be lost; once resources become available, your agents can pick it up.

So if the semantic you’re looking for is not losing work by maintaining a queue of tasks, Prefect will work for you. If you’re looking to build a queueing system with direct control of the queue itself, you may prefer a different framework (or to use your own queue to trigger a Prefect job via API)

<@UKTUC906M>  Thanks a lot for the detailed reply. Yea, I don't need direct control of the queue.
I have a web app with flow as follows: downloading data from user, running it through a bunch of machine learning models and then generating graphs and stuff.
Problem I was facing right now is that if more than 5 instances of this flow run concurrently, it crashes the server. I think prefect can help me out here.

Hi Sumant, yes, I think Prefect can help. In the near future, we’ll be adding more features around limiting flow concurrency through the API, which would help you even further in this instance (you could, for example, only allow 1 flow to run at a time)