https://prefect.io logo
s

scott

08/10/2023, 10:24 PM
Any discussion on the benefits of push work pools? I get that you then don’t need to run workers or agents, but there’s likely tradeoffs, and if so what are they? When would you use push vs. pull?
j

Jake Kaplan

08/11/2023, 9:03 PM
Hey there! For benefits of push pools, the main one is for sure its not needing a worker. This helps in a few ways though: • it’s one less thing you need to deploy and monitor yourself to get your flows running • you don’t need to worry about scaling out multiple workers for lots of runs, prefect cloud is effectively assuming that responsibility • It’s also way easier to create and use multiple pools if you don’t need to go through the process of deploying a new worker to read from the new pool • a small one but: push pools pick up your flow runs faster. You may notice sometimes that your flow run gets marked as momentarily late depending on where the worker is in it’s poll cycle and the latency to poll the api. push pools let the run move to pending almost immidiately For tradeoffs: • it’s only available for a subset of infrastructure types (ecs, cloud run, aci) • You’re required to use a credentials block. If you were to host the worker, that machine might already have some perms to deploy flow runs in your infrastructure. • Push pools are still in beta and don’t support priority or pool level concurrency atm (ideally though push pools lets you scale where priority is maybe less important depending on your use case) In terms of when I would use push vs pull (I am likely a little biased 😁 ) but if you don’t have security restrictions that keep you from saving credentials blocks with prefect cloud, I would always default to push pools. It cuts out a ton of friction.
👍 1
happy to answer any more specific questions around push pools if you have any
s

scott

08/11/2023, 9:05 PM
Great, thanks very much!