i

    Isaac Brodsky

    9 months ago
    Ever seen this error in Kubernetes?
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"dask-root-elided\" is forbidden: User \"system:serviceaccount:default:default\" cannot get resource \"pods/log\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"name":"dask-root-elided","kind":"pods"},"code":403}
    Was trying to install a new Prefect agent and it seems the RBAC setup is not right? The agent was configured with
    prefect agent kubernetes install --key $PREFECT_API_KEY --rbac --label my_label_here
    Prefect version 0.15.9 Running on GKE
    Anna Geller

    Anna Geller

    9 months ago
    @Isaac Brodsky for Dask, the role is a bit different, check out this blog post that shows how to set up the Role and Role Binding: https://medium.com/slateco-blog/prefect-x-kubernetes-x-ephemeral-dask-power-without-responsibility-6e10b4f2fe40
    i

    Isaac Brodsky

    9 months ago
    Ok, I'll give that RBAC setup a try
    Anna Geller

    Anna Geller

    9 months ago
    To explain why: you need permissions to create and interact with a temporary Dask cluster
    i

    Isaac Brodsky

    9 months ago
    If you can add that to the production agent install instructions in the Prefect docs that would be really helpful
    I think also the setup for running jobs in a separate namespace needs some additional RBAC config
    Confirming that was the missing RBAC config, thanks @Anna Geller