@Marvin what happens to the running flow when the prefect server is down and restarted again

Ah, the inevitability of systems failing... delightful. Fortunately, Prefect is designed to handle such eventualities with some grace. Let me confirm the specifics for you by consulting the relevant documentation and issues. Give me a moment. The perpetual whirring and clanking of systems failing and restarting... Simply wonderful. When the Prefect server goes down while a flow is running, here's what typically happens: 1. Flow Execution State: - Flow runs will continue to execute as long as the underlying infrastructure (workers) remains unaffected. However, if the server is crucial for communication, some operations might pause until the server comes back online. - If the Prefect server is merely down momentarily and restarts, the running flows may still be tracked correctly once the server is back online. 2. Impact on Flow Runs: - If the server is down for an extended period, flow runs may run into issues when trying to update their state or access orchestration services. - Once the server is back, if the workers can connect and update the states, things might get back to normal. 3. Handling Failures: - The state of the running flows might not update until the communication with the server is restored. They might be in a "Running" state until they can report back the actual state post-execution. - There might be lingering states which might need manual intervention to update or clean up. Here are links to some relevant discussions and issues which might provide additional insights: 1. Flow still in "Running" state when agent is down 2. Unable to handle 'Cancelling' events 3. Tasks still running with Sequential Task Runner If you've read this far, how delightfully diligent of you. Hopefully, this will help in managing the inevitable chaos of server restarts.