Hi All, Cloud Run Timeout Not Reported as Flow Failure I'm using cloud run as a work-pool. I set the cloud run timeout to 30 mins. My flow runs for more than 30 mins. On logs we see that job failed - timeout, but the Flow continued on

Running

state forever (till I manually canceled it). Can someone advise on how to have the flow to be marked as failed in this case? Is it by design?

Encountered an exception while waiting for job run completion - Timeout of 1800 seconds reached while waiting for Cloud Run Job V2 prefect-polar-skylark to complete.
02:00:15 PM
prefect.flow_runs.worker
An error occurred while monitoring flow run '425154ed-1111-1111-1111-4b3cf987ad0b'. The flow run will not be marked as failed, but an issue may have occurred.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/workers/base.py", line 896, in _submit_run_and_capture_errors
    result = await self.run(
  File "/usr/local/lib/python3.10/site-packages/prefect_gcp/workers/cloud_run_v2.py", line 423, in run
    result = await run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/prefect_gcp/workers/cloud_run_v2.py", line 694, in _watch_job_execution_and_get_result
    execution = self._watch_job_execution(
  File "/usr/local/lib/python3.10/site-packages/prefect_gcp/workers/cloud_run_v2.py", line 776, in _watch_job_execution
    raise RuntimeError(
RuntimeError: Timeout of 1800 seconds reached while waiting for Cloud Run Job V2 prefect-polar-skylark to complete.

Hi Isaac, can you share a little bit about how you set the timeout? Did you set

timeout_seconds

as an argument in the flow decorator?

HI Bianca, the timeout is set on the prefect.yaml file.

prefect.prod.yaml

deployments:
  - name: my-flow
    description: null
    entrypoint: my_path_inside_docker/my_file.py:my_flow
    flow_name: null
    parameters: { }
    schedule:
      cron: "*/10 * * * *"
      timezone: UTC
    tags:
      - prod
    work_pool:
      name: my-cloud-run-work-pool
      job_variables:
        image: *image_prod
        service_account_name: "<mailto:my_account@my-project.iam.gserviceaccount.com|my_account@my-project.iam.gserviceaccount.com>"
        timeout: 1800
        labels:
          flow_name: "my_flow_name"
          env: "prod"

Hey again! just wanted to let you know that we were able to reproduce this. (new issue here)

Hi Bianca, thanks for handling the issue. The prefect version is

prefect = "^2.13.8"