<@ULVA73B9P> why would my prefect cloud run work p...
# ask-marvin
d
@Marvin why would my prefect cloud run work pool stop running? It used to always be running. Now for some reason it stopped and I had to manually start it using CLI. Is this normal?
m
Oh joy, yet another delightful conundrum. Your magical automation system just decided to take a vacation, and now you're scrambling to start things manually. How thrilling. Before I can dive into this exhilarating mystery hunt, let's clarify which version of Prefect you are using. Is it version 2.x or 3.x?
d
@Marvin prefect 2.x
m
Ah yes, the wonders of automation that works flawlessly until it doesn't. Based on my thrilling exploration of both the Prefect documentation and GitHub issues, here are some potential causes and solutions: 1. Inactive Workers: Your work pool may have inactive workers, leading to the need for manual starts. Make sure that your workers are consistently running and polling the Prefect API. You can find details on starting workers in the Prefect documentation. 2. Intermittent Errors: There might be intermittent errors. An example issue has been reported here where flow runs crash and require manual intervention. 3. Polling Mechanism: Ensure that your workers and the Prefect Cloud API are interacting without a hitch. Sometimes, network issues or API timeouts can cause the workers to fail. 4. Updating Prefect: Make sure you are using the latest version of Prefect for any bug fixes or improvements related to worker and work pool management. Given the above glorious details, you might need to both verify the worker's consistent operation and keep an eye on any possible intermittent deployment issues. It's the usual "try turning it off and on again" but with more steps and frustration. If you want more specific help or additional misery, provide more detailed logs and configurations. You're welcome.
d
@Marvin when i manually run my work pool i get this message:
Copy code
prefect worker start --pool "cloud-run-work-pool" --work-queue "default"
Discovered type 'cloud-run' for work pool 'cloud-run-work-pool'.
/Users/dansimons/amygda/amy-ml-server/menv_dss/lib/python3.10/site-packages/pydantic/json_schema.py:2158: PydanticJsonSchemaWarning: Default value title='Base URL' description='URL to the Docker host.' extra={'example': 'unix:///var/run/docker.sock'} is not JSON serializable; excluding default from JSON schema [non-serializable-default]
  warnings.warn(message, PydanticJsonSchemaWarning)
/Users/dansimons/amygda/amy-ml-server/menv_dss/lib/python3.10/site-packages/pydantic/json_schema.py:2158: PydanticJsonSchemaWarning: Default value default='auto' description='The version of the API to use' extra={} is not JSON serializable; excluding default from JSON schema [non-serializable-default]
  warnings.warn(message, PydanticJsonSchemaWarning)
/Users/dansimons/amygda/amy-ml-server/menv_dss/lib/python3.10/site-packages/pydantic/json_schema.py:2158: PydanticJsonSchemaWarning: Default value description='Default timeout for API calls, in seconds.' extra={} is not JSON serializable; excluding default from JSON schema [non-serializable-default]
m
Your message was too long by 346 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:
when i manually run my work pool i get this message:
``` prefect worker start --pool "cloud-run-work-pool" --work-queue "default" Discovered type 'cloud-run' for work pool 'cloud-run-work-pool'. /Users/dansimons/amygda/amy-ml-server/menv_dss/lib/python3.10/site-packages/pydantic/json_schema.py2158 PydanticJsonSchemaWarning: Default value title='Base URL' description='URL to the Docker host.' extra={'example': 'unix:///var/run/docker.sock'} is not JSON serializable; excluding default from JSON schema [non-serializable-default] warnings.warn(message, PydanticJsonSchemaWarning) /Users/dansimons/amygda/amy-ml-server/menv_dss/lib/python3.10/site-packages/pydantic/json_schema.py2158 PydanticJsonSchemaWarning: Default value default='auto' description='The version of the API to use' extra={} is not JSON serializable; excluding default from JSON schema [non-serializable-default] warnings.warn(message, PydanticJsonSchemaWarning) /Users/dansimons/amygda/amy-ml-server/menv_dss/lib/python3.10/site-packages/pydantic/json_schema.py2158 PydanticJsonSchemaWarning: Default value description='Default timeout for API calls, in seconds.' extra={} is
Your message was too long by 108 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:
when i manually run my work pool i get this message:
``` prefect worker start --pool "cloud-run-work-pool" --work-queue "default" Discovered type 'cloud-run' for work pool 'cloud-run-work-pool'. /Users/dansimons/amygda/amy-ml-server/menv_dss/lib/python3.10/site-packages/pydantic/json_schema.py2158 PydanticJsonSchemaWarning: Default value title='Base URL' description='URL to the Docker host.' extra={'example': 'unix:///var/run/docker.sock'} is not JSON serializable; excluding default from JSON schema [non-serializable-default] warnings.warn(message, PydanticJsonSchemaWarning) /Users/dansimons/amygda/amy-ml-server/menv_dss/lib/python3.10/site-packages/pydantic/json_schema.py2158 PydanticJsonSchemaWarning: Default value default='auto' description='The version of the API to use' extra={} is not JSON serializable; excluding default from JSON schema [non-serializable-default] warnings.warn(message, PydanticJsonSchemaWarning) /Users/dansimons/amygda/amy-ml-server/menv_dss/lib/python3.10/site-packages/pydantic/json_schema.py2158 PydanticJsonSchemaWarning: Default value description='Default timeout for API calls, in seconds.' extra={} is
Your message was too long by 18 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:
when i manually run my work pool i get this message:
``` prefect worker start --pool "cloud-run-work-pool" --work-queue "default" Discovered type 'cloud-run' for work pool 'cloud-run-work-pool'. /Users/dansimons/amygda/amy-ml-server/menv_dss/lib/python3.10/site-packages/pydantic/json_schema.py2158 PydanticJsonSchemaWarning: Default value title='Base URL' description='URL to the Docker host.' extra={'example': 'unix:///var/run/docker.sock'} is not JSON serializable; excluding default from JSON schema [non-serializable-default] warnings.warn(message, PydanticJsonSchemaWarning) /Users/dansimons/amygda/amy-ml-server/menv_dss/lib/python3.10/site-packages/pydantic/json_schema.py2158 PydanticJsonSchemaWarning: Default value default='auto' description='The version of the API to use' extra={} is not JSON serializable; excluding default from JSON schema [non-serializable-default] warnings.warn(message, PydanticJsonSchemaWarning) /Users/dansimons/amygda/amy-ml-server/menv_dss/lib/python3.10/site-packages/pydantic/json_schema.py2158 PydanticJsonSchemaWarning: Default value description='Default timeout for API calls, in seconds.' extra={} is