<@ULVA73B9P> Explain to me how automations work in...
# marvin-ai
l
@Marvin Explain to me how automations work in Prefect, are they using the usual work pool queue or how are the workers picking up the work?
m
thought for 175.2 seconds
Great question. Short answer: automations don’t run your flows themselves—they react to events and, when appropriate, create flow runs that are picked up by your existing work pool/worker setup. Here’s how it works in Prefect 3.x: - What automations are - Automations are event-driven “if this, then that” rules managed by Prefect Cloud. They listen to events (e.g., a flow-run fails, completes, or a custom webhook fires) and execute actions (e.g., run a deployment, send a notification, cancel a run). - They are separate from workers and work pools; they don’t execute code. - How work actually gets picked up - When an automation uses the RunDeployment action, Prefect Cloud creates a new flow run for that deployment. - That flow run is queued onto the deployment’s configured work pool (and queue, if you use queues). - Your workers, which are already polling the relevant work pool/queue, discover the run and execute it just like any other scheduled run. - Other actions (e.g., SendNotification, CancelFlowRun, ChangeFlowRunState) don’t involve workers at all. - Automations vs schedules - Schedules are time-based and proactively create runs on a cadence. - Automations are event-based and react to things that did (reactive) or did not (proactive) happen within a time window. - Typical flow 1) An event occurs (e.g.,
prefect.flow-run.Completed
on an upstream deployment). 2) Your automation’s trigger matches that event and its conditions (threshold/window) are satisfied. 3) The automation runs its action—often RunDeployment—which creates a new flow run. 4) The new flow run is placed in the deployment’s work pool/queue and is picked up by workers polling that pool/queue. Helpful docs: - Concepts: Event triggers - How-to: Creating automations - Chaining flows: Chaining deployments with events - Deployment triggers (define trigger logic alongside your deployment): Creating deployment triggers If you share your use case (e.g., chaining deployments, SLA monitoring, error handling), I can suggest a concrete trigger/action setup.
l
@Marvin Usually the workers pool every 15s for example to check the queue, but it seems that with automaitons the flow gets scheduled immediately, how so?
m
thought for 206.2 seconds
Great observation. What you’re seeing is the difference between “creating/scheduling a flow run” and “a worker picking it up to execute.” - Automations act immediately - When an automation fires (e.g., RunDeployment), Prefect Cloud creates the flow run right away and places it in the deployment’s work pool/queue. That’s why you see “Scheduled” in the UI immediately—no worker is involved in this step. - Workers still poll to pick up runs - Workers poll their work pool/queues on a fixed interval to find runs that are ready. In Prefect 3.x the default polling interval is typically ~10s (configurable), not long-polling. - There’s also a prefetch window: workers will pick up runs whose scheduled_time is within “now + prefetch_seconds” (defaults to ~10s). So if the automation creates a run scheduled for “now,” a worker will grab it on its next poll. Put together: - Automations create the flow run immediately (visible as SCHEDULED). - The actual start time is gated by the worker’s next poll plus any prefetch behavior. If you want even faster pickup, you can: - Reduce the worker’s polling interval. - Adjust the worker’s prefetch window so “now” runs are eligible sooner. Helpful refs: - Workers: Workers - Work pools: Work pools - Automations and event triggers: Event triggers, Creating automations - Settings reference (polling/prefetch): Settings reference If you share your worker type and how you’re starting it, I can point to the exact flag/env vars to tune the poll and prefetch settings.
n
the technical difference here is that there’s an events firehose (websockets) that you can tap into server side vs workers which poll every X seconds
👀 1
l
So actually the answer from Marvin is wrong? It seems that in this case the worker is not polling the queue but the task is given to it somehow, could you elaborate a bit on how this works?
n
i think marvin’s answer is correct. there are 2 things happening: • websocket based background services run server side which stream in events and evaluate automation triggers. if a trigger fires, the action takes place, ie the run is scheduled • client side, your worker is polling the server for scheduled runs. once this RunDeployment action schedules the run, the worker will find it in its next poll
👍 1
l
Thanks!