Thanks for the detailed traces — that helps a lot. You’re hitting two separate issues: 1) AttributeError: Runner object has no attribute _runs_task_group - Diagnosis: This is a Prefect 3.x bug in the flow runner’s cancellation/observer logic. The code path that handles “on_cancelling” tries to reference

_runs_task_group

but the internal attribute is

_loops_task_group

. It shows up during cancellation or cleanup and can surface even if the flow otherwise runs. - Fix: Upgrade Prefect in your flow image and your worker image to the latest 3.x. We’ve seen this resolved by moving to a newer 3.x release (current is 3.6.1). Be sure the server, worker, and flow runtime all use compatible/newer versions. Example upgrade steps in your images:

pip install --upgrade 'prefect>=3.6.1'
# if you’re using integrations, upgrade them too
pip install --upgrade 'prefect-gcp>=0.6.0' google-api-python-client

After upgrading, rebuild/redeploy your Cloud Run job image(s) and restart the worker. Useful links: - Prefect release notes: Prefect 3.x release notes 2) googleapiclient HttpError in Cloud Run V2 ExecutionV2.get (while monitoring the job) - Diagnosis: This occurs when the Cloud Run V2 worker polls the Cloud Run Jobs API to monitor execution. The most common causes are: - Insufficient IAM on the worker’s service account to read executions - Wrong project/region configuration - Cloud Run API not enabled in the project - Credentials not being picked up correctly inside the worker container - Recommended checks/fixes: 1) Ensure the correct APIs are enabled in the target GCP project: - Cloud Run API (run.googleapis.com) 2) Verify the worker’s service account IAM. The worker SA needs to be able to create and monitor Cloud Run Job executions. At minimum: - roles/run.viewer (to read executions) — for monitoring only - If the worker is creating executions (it is), give it roles/run.admin - If the Cloud Run Job runs as a different service account, grant the worker SA roles/iam.serviceAccountUser on the job’s runtime SA 3) Confirm project and region are correct in your Cloud Run V2 worker variables: - The region in your worker configuration must match the region where the job executes (e.g., us-central1) 4) Ensure credentials are correctly available to the worker: - On Cloud Run Jobs, prefer using the job’s attached service account with the roles above - If using a key, confirm the JSON is mounted and GOOGLE_APPLICATION_CREDENTIALS points to it 5) Get the exact HttpError details: - The

googleapiclient.http.HttpError

includes an HTTP status code and message (e.g., 401/403/404). That narrows the root cause precisely: - 401: credentials invalid/unavailable - 403: credentials valid but missing permission (add roles as above) - 404: region/project/job name mismatch or resource not found 6) Quick validation outside Prefect (with the same SA):

gcloud run jobs executions describe --region <region> <job-name> --project <project-id>

- If this fails with the worker’s SA, it’s an IAM or region/project issue. Docs: - Cloud Run v2 worker guide: Cloud Run V2 worker guide Next steps / what I need from you to confirm: - Which Prefect versions are in: - Prefect server (self-hosted API/Server) - Worker image(s) - Flow runtime image - Your prefect-gcp version - The full

HttpError

status code and message body from the Cloud Run monitoring failure - The IAM roles on the service account used by the Prefect Cloud Run V2 worker, and (if different) the SA used to run the Cloud Run Job In parallel, I recommend: - Upgrading Prefect and prefect-gcp as shown above and redeploying the images - Verifying the worker SA has roles/run.admin and roles/iam.serviceAccountUser on the job’s runtime SA

- Ensuring the region and project values in your worker configuration match the target Cloud Run Job If you share the exact HttpError code/message and your version/IAM specifics, I can pinpoint the minimal IAM needed and confirm whether anything else needs adjusting.