@Marvin I have a prefect server running with several work processes. They are all working correctly and I can run flows on the workers. The deployments are set to run every 10 minutes. When i restart the server, sometimes some of the workers come back (as in they show up again in the work pools), but usually not all of them.The only way to get them all back is to restart the workers themselves. Is this expected behaviour?

It seems that the behavior you're experiencing with Prefect workers not reconnecting after a server restart is not explicitly documented as expected behavior. Prefect workers are designed to poll work pools for new runs to execute, and they should ideally reconnect to the server after a restart if the environment is set up correctly. Here are a few things to consider: 1. Environment Configuration: Ensure that the environment where your workers are running has the correct

PREFECT_API_URL

and

PREFECT_API_KEY

settings. These are necessary for the workers to connect to the Prefect server. 2. Worker Persistence: Workers are lightweight polling services, and they should be set up to start automatically with the server. If this isn't happening, you might need to configure your environment to ensure workers restart alongside the server. 3. Health Checks: Consider enabling the

--with-healthcheck

option when starting your workers. This could help in diagnosing if the workers are failing to reconnect due to any health issues. 4. Manual Restart: As a workaround, restarting the workers manually, as you've been doing, is a temporary solution until a more robust setup is configured. For more detailed information, you can refer to the Prefect documentation on workers. If this continues to be a problem, consider reaching out to the Prefect community or filing a bug report, as there are no open issues related to this behavior in the Prefect GitHub repository.