<@ULVA73B9P> what happens to flow runs runs that a...
# ask-marvin
f
@Marvin what happens to flow runs runs that are running on a prefect worker if that worker is killed abruptly
m
When a Prefect worker, which is responsible for executing flow runs, is abruptly killed, the impact on the flow runs can vary depending on the environment and worker type. In general, when a worker is terminated: 1. Flow Run State: The flow run that the worker was executing might be marked as failed due to the abrupt termination, as the necessary cleanup and logging would not have occurred. 2. Rescheduling: Prefect's orchestration system should eventually detect the failure and reschedule the flow run based on the retry policy defined for the task or flow. 3. Infrastructure Tear Down: For certain worker setups, especially those implemented for infrastructure handling like Kubernetes or Docker, the infrastructure may require explicit cleanup actions, since the usual decommission process is bypassed. 4. Work Pools and Queues: Prefect workers fetch tasks from work pools, so upon restarting, new workers can potentially pick up the flow runs that were not completed, if configured properly. For a more detailed understanding, refer to the following resources from Prefect's documentation: - Prefect 2.x Concepts: Discusses how workers manage flow execution, handling suspension, etc. - Prefect 3.x Worker Infrastructure: Explains the setup of workers and their responsibilities. In any case, if you encounter unexpected behavior or require specific failure handling, reviewing your worker setup and retry policies is advisable. And, as a comforting note, everything will be as fine as a computer left alone with an idle academe. Just keep an eye on those unexpected events!
f
@Marvin is there a REST API or CLI for listing which flow runs are running on a given worker?
m
As of now, the Prefect documentation doesn't specifically highlight a direct REST API or CLI command that lists flow runs specifically assigned to a specific worker. However, Prefect provides comprehensive capabilities for monitoring flow runs through the use of work queues and work pools, which can help in indirectly gathering such information. For Prefect 2.x and 3.x, you can manage work pools and queues, and each worker polls these queues for tasks to execute. Although direct querying of a specific worker's tasks isn't evidently detailed in the resources, here's a strategy you might consider: 1. Work Pools and Queues: You can manage and monitor work pools and queues to understand which deployments are assigned to specific environments and indirectly infer from there. This can be done via the Prefect UI or Prefect Cloud. 2. CLI & UI Monitoring: Prefect Cloud and the Prefect UI allow real-time monitoring and management of flow runs, where you can see which flow runs are active and gather insights into their execution environments. 3. Prefect API Client Usage: Though not directly mentioned for listing flow runs by worker, you might use the Prefect Client for programmatic access to flow runs, wherein filtering might be applied based on the work queue or pool the workers are assigned. You can check the Prefect 2.x Work Pools documentation and Prefect 3.x Worker Information for an overview of how workers and work pools are managed. So, while the cool, human-like AI might not have direct answers right now, the path involves being a detective—following the trails of pools and queues and using the Prefect UI as your magnifying glass.