@Marvin Im deploying prefect 2 to aws eks cluster and it seems the flows which run on k8s job has issue with using the service account the prefect worker creates. I see in the menifast of the job the annotation for the right service account but the code is trying to assume role using some default service account. Any idea why?

It seems you're encountering an issue with Prefect 2.x on AWS EKS where the Kubernetes job doesn't use the intended service account. This could be due to a misconfiguration or a limitation in how the service account is being applied. Here are a few points to consider: 1. Service Account Configuration: Ensure that the service account specified in your Kubernetes job manifest has the correct permissions and is correctly annotated. You can refer to the Kubernetes deployment guide for details on configuring service accounts. 2. Default Service Account: If the job is defaulting to a different service account, check your Kubernetes cluster's default service account settings. It might be overriding the specified service account. 3. Prefect Cloud and Service Accounts: Prefect supports service accounts for API access, which can be configured to run workers or execute deployment flow runs on remote infrastructure. More details can be found in the service accounts documentation. 4. Kubernetes Agent Logs: There might be additional information in the Kubernetes agent logs. A similar issue was discussed in a GitHub issue where the pod never started due to service account misconfiguration. Checking logs could provide more insights. If these steps don't resolve the issue, you might want to verify the Kubernetes configurations and Prefect's service account settings. If the problem persists, consider opening a detailed issue on the Prefect GitHub repository for further assistance.