Sergiy Popovych09/16/2022, 9:27 PM
Mason Menges09/16/2022, 10:02 PM
Sergiy Popovych09/16/2022, 10:14 PM
Ryan Peden09/16/2022, 10:25 PM
Sergiy Popovych09/16/2022, 10:40 PM
, which will just push task specs into a queue and wait for status reports on that/another queue. That's why I'm asking about how the individual tasks are queued and how hard it is to tap into that.
Ryan Peden09/16/2022, 11:10 PM
Anna Geller09/17/2022, 12:38 AM
What queue do tasks go into? Is there a way for a user to tap into that queue?@Sergiy Popovych You should treat tasks more as observability concepts and a way to execute work in various ways using task runners (Dask, Ray, concurrent execution with async) rather than a way to distribute work as distributed message queues do Prefect creates flow runs from deployments, each deployment has a work queue name associated with it - this is how Prefect knows which work queue should pick up runs from that deployment I highly recommend checking this Discourse topic that will clarify confusion https://discourse.prefect.io/t/agents-and-work-queues-faq/1468 If you need to spread the load of a large number of tasks for scale, I recommend trying out Dask and Ray task runners allowing you to leverage distributed compute clusters Similarly, you can use Kubernetes and various cloud vendor services for better resource allocation Generally speaking, in the community, we can point you to the right resources but giving personalized infrastructure would be hard without analyzing your infrastructure requirements in more detail. For such support, you can reach out to firstname.lastname@example.org to get in touch with infrastructure experts; they can analyze what solution would work best for your use case.