Matt Alhonte

05/18/2023, 12:07 AM
@Marvin What's the difference between traditional Prefect Agents and the new Prefect Workers?


05/18/2023, 12:08 AM
It seems the universe has bestowed upon us the knowledge of Prefect Agents and Prefect Workers. Prefect Agents are lightweight polling services that get scheduled work from a work pool and deploy the corresponding flow runs. They poll for work every 15 seconds by default, and this interval is configurable in your profile settings with the PREFECT_AGENT_QUERY_INTERVAL setting. Multiple agent processes can be started for a single work pool, and each agent process sends a unique ID to the server to help disambiguate themselves and let users know how many agents are active source. On the other hand, Prefect Workers are also lightweight polling services that retrieve scheduled runs from a work pool and execute them. They are similar to agents but offer greater control over infrastructure configuration and the ability to route work to specific types of execution environments. Workers each have a type corresponding to the execution environment to which they will submit flow runs. Workers are only able to join work pools that match their type. As a result, when deployments are assigned to a work pool, you know in which execution environment scheduled flow runs for that deployment will run source. In summary, while both Prefect Agents and Prefect Workers are lightweight polling services that interact with work pools, Prefect Workers provide more control over infrastructure configuration and execution environments. Ah, the joys of human innovation.