<@ULVA73B9P> I have a script that works correctly ...
# ask-marvin
c
@Marvin I have a script that works correctly when I run it locally, but when I run it from my Kubernetes cluster, I receive the following error:
Copy code
Task run failed with exception: ResourceNotFoundError('The specified container does not exist.\nRequestId:84e44cab-801e-005c-71dc-623316000000\nTime:2025-12-01T16:09:16.5744514Z\nErrorCode:ContainerNotFound') - Retries are exhausted
m
thought for 74.8 seconds
Thanks for the details. That error is coming from Azure Blob Storage — “container” here means an Azure Blob container (not a Kubernetes/Docker container). It typically means the code running in Kubernetes is pointing at a different storage account/container, or it’s missing credentials/permissions to read it. Common causes when it works locally but fails in-cluster: - Different config in the pod: environment variables for account name, container name, connection string, or SAS aren’t set in the Kubernetes job/pod. - Using a different storage account/tenant/subscription in-cluster (e.g., different connection string, secret, or managed identity). - The container is created locally (dev code path) but not created in the cluster (prod code path). - Name mismatch or casing: Azure container names must be lowercase and immutable; string interpolation/env expansion in-cluster may be producing a different name (e.g., empty or suffixed). - Permissions/auth: the pod’s identity/credentials lack Blob Data Reader/Contributor for that container. - Less likely but possible: storage firewall/private endpoints causing requests to another account/endpoint. Quick checks to run: 1) Log what your code is actually using in-cluster - Log these at runtime in the pod: storage account, container name, and the full endpoint you build. - Confirm they match your local values exactly. 2) Verify credentials are present in the pod - If you use a connection string: ensure the env var is set on the Kubernetes job/pod. - If you use account name/key: ensure both are present. - If you use SAS: verify it hasn’t expired and grants the right container scope. - If you use AKS Workload Identity/Managed Identity: confirm the identity is bound to the pod and has Storage Blob Data Reader/Contributor on the container. 3) Test from inside the running pod Add a small diagnostic step (run once) to list containers and check the target exists with the same credentials your task uses:
Copy code
import os
   from azure.storage.blob import BlobServiceClient

   def debug_azure():
       container = os.getenv("AZURE_BLOB_CONTAINER")
       conn = os.getenv("AZURE_STORAGE_CONNECTION_STRING")
       acct = os.getenv("AZURE_STORAGE_ACCOUNT_NAME")
       key = os.getenv("AZURE_STORAGE_ACCOUNT_KEY")

       print(f"Resolved container={container!r}, account={acct!r}")

       if conn:
           svc = BlobServiceClient.from_connection_string(conn)
       elif acct and key:
           svc = BlobServiceClient(account_url=f"https://{acct}.<http://blob.core.windows.net|blob.core.windows.net>", credential=key)
       else:
           raise RuntimeError("No Azure credentials found (connection string or account/key).")

       print("Containers visible from this pod:")
       for c in svc.list_containers():
           print(" -", c['name'])

       # Will raise if the container truly doesn't exist or is inaccessible
       svc.get_container_client(container).get_container_properties()
       print("Container exists and is accessible from the pod.")
4) Ensure the container creation behavior is consistent - If your local run creates the container when missing, ensure the same code path and permissions are used in-cluster. If you rely on pre-created infra, confirm the container exists in the exact target account. 5) Prefect/Kubernetes specifics - If you’re using a Prefect Kubernetes work pool: make sure your job template (or deployment) passes the required env vars to the flow’s pod. - If you’re using private endpoints or storage firewall rules: confirm the cluster can resolve/reach the blob endpoint and is allowed by the storage account network rules. If you can share a bit more, I can be more specific: - Prefect version (assuming 3.x unless you say otherwise) and how you’re running (Kubernetes work pool? Custom job template?). - How you authenticate to Azure from the pod (connection string, account key, SAS, workload identity).
- The exact code snippet that constructs the BlobServiceClient and container client. - The values you expect for storage account and container name, and what the pod resolves at runtime (sanitized is fine). Once we know those, we can pinpoint whether it’s a config/secret propagation issue, a name mismatch, or a permissions/network problem.