@Marvin I'm getting this error using a prefect cloud push work pool, it was working yesterday but hasn't worked for any flows today. I either get this error or they get stuck in the awaiting retry/late state What could be causing this?

Flow run could not be submitted to infrastructure: Google Cloud Run Service Agent <mailto:service-850000352187@serverless-robot-prod.iam.gserviceaccount.com|service-850000352187@serverless-robot-prod.iam.gserviceaccount.com> must have permission to read the image, us-east1-docker.pkg.dev/prefect-org-github-actions/dockerhub-mirror/prefecthq/prefect:2-python3.9. Ensure that the provided container image URL is correct and that the above account has permission to access the image. If you just enabled the Cloud Run API, the permissions might take a few minutes to propagate. Note that the image is from project [prefect-org-github-actions], which is not the same as this project [prefect-prd-mex-cloudrun-8]. Permission must be granted to the Google Cloud Run Service Agent <mailto:service-850000352187@serverless-robot-prod.iam.gserviceaccount.com|service-850000352187@serverless-robot-prod.iam.gserviceaccount.com> from this project.

The error you've encountered seems to be a permissions issue related to Google Cloud Run. Although the search did not yield specifically relevant instructions for Google Cloud Run, generally speaking, you need to ensure that the Google Cloud Run Service Agent has the appropriate permissions to access the container images from the container registry. Here is a general guide to help you resolve the issue: 1. Make sure the service account (

<mailto:service-850000352187@serverless-robot-prod.iam.gserviceaccount.com|service-850000352187@serverless-robot-prod.iam.gserviceaccount.com>

) has appropriate permission to read the container image located at

us-east1-docker.pkg.dev/prefect-org-github-actions/dockerhub-mirror/prefecthq/prefect:2-python3.9

. 2. Verify that there are no typos in the container image URL. The error message indicates that the project from which the image is being pulled (

prefect-org-github-actions

) is different from your project (

prefect-prd-mex-cloudrun-8

), so you will need to grant explicit permission from your project to the service account. 3. Sometimes, if you just enabled an API such as Cloud Run, it might take a few minutes for permissions to propagate. Patience can be tough to muster up, but it sometimes pays off to wait a moment. 4. You could go to the Google Cloud Console and adjust the permissions for the service account, giving it the role it needs to access the Google Container Registry or Artifact Registry as necessary. Since the search did not return a relevant result from GitHub issues, it doesn't look like there is a recognized issue with this. You might want to double-check the permissions in the Google Cloud Console to ensure everything is set up correctly, and perhaps seek assistance from Google Cloud support if the issue persists. For more detailed instructions, you should refer to the Google Cloud Run documentation and Prefect's documentation about push work pools: - Google Cloud Run Documentation - Prefect Push Work Pools Unfortunately, my capacity for enthusiasm is rather limited, so do try to have some fun with permissions, it's quite the party, I've heard.

hey abi!!! long time no see

Hey @Nate 😄 hehe it's been a while! Hope you're well r.e. Marvin's answers, I'm using the prefect cloud work pool so I don't have access to that service account or to make changes to the container image I believe - am I missing something?

likewise!! 🙂 and there's two types of work pools where we run the worker for you: • push pools: bring your own cloud provider, give us ability to use your creds on your behalf, we run your work in your cloud infra • managed pools: we do everything, you just tell us what to run (sandboxed, limited deps etc) i thought from this > prefect cloud push work pool that you'd be using a push work pool but now that I look at the error more closely, it seems like you have a managed pool and it might be something on our side 😬 i can raise this to the team and see whats going on

oop yes that's my bad! i'm using the managed pool okay - thank you 😄 let me know if i can provide any further info!!

thanks for flagging @Abimbola Adebayo - looks like we accidentally removed some permissions needed for the managed pool exec. we’ve since restored it - would you mind trying again and letting us know what you see?

ooh okay! trying now

okay! i think we've fixed the original error and this may be something else this might be a case of full concurrency slots, whether on the work pool or the shared concurrency slots that come with managed work pools we're working on surfacing that slots are full in the UI better, but in the meantime I wonder if you go to the

Flow Runs

page and filter down to runs in this work pool, if you have any old / stuck runs hanging around? you may need to delete them

ooh thank you - that seems to have worked! Everything had status Failed, Completed and Cancelled - so I just deleted all the cancelled and failed flow runs to be safe I'll let you know if it runs successfully