Benjamin R. Everett
09/04/2025, 11:09 PMMarvin
09/04/2025, 11:09 PMMarvin
09/04/2025, 11:11 PMPREFECT__CLOUD__AGENT__AUTH_TOKEN
. That was a Prefect 1.x variable. In Prefect 2.x you authenticate using API keys via PREFECT_API_KEY
(and your workspace’s PREFECT_API_URL
) or by running prefect cloud login
.
How to do it in Prefect 2.x:
1) Create an API key in Prefect Cloud
- In the Cloud UI, go to your user menu → API Keys → “Create API Key”
- For automation, consider creating a Service Account and a key for it
- Docs: API keys and Service accounts
2) Authenticate your environment
- Option A: CLI login (creates/uses a Prefect profile)
prefect cloud login --key <YOUR_API_KEY> --workspace <account_handle>/<workspace_handle>
- Option B: Environment variables (useful in containers)
export PREFECT_API_KEY=<YOUR_API_KEY>
export PREFECT_API_URL="<https://api.prefect.cloud/api/accounts/<ACCOUNT_ID>/workspaces/<WORKSPACE_ID>>"
You can copy the account/workspace IDs from your workspace settings in the UI.
3) Start your worker (recommended in 2.x)
prefect worker start -p <your-work-pool-name>
Docs: Workers concept
Notes:
- If you see references to PREFECT__CLOUD__AGENT__AUTH_TOKEN
, that’s a Prefect 1.x pattern. In 2.x, both workers (recommended) and legacy agents use PREFECT_API_KEY
for Cloud auth.
- API keys are only shown once—store them securely.
If you’re actually on Prefect 1.x and need that exact token, let me know and I’ll share the v1 steps. Otherwise, what environment are you authenticating (local, VM, container, Kubernetes)? I can tailor the exact commands.Benjamin R. Everett
09/04/2025, 11:18 PMBenjamin R. Everett
09/04/2025, 11:19 PMMarvin
09/04/2025, 11:19 PMMarvin
09/04/2025, 11:22 PMPREFECT_API_KEY
).
High-level steps
1) Create an API key in Prefect Cloud
- Cloud UI → User menu → API Keys → Create
- For automation, consider a Service Account + key
- Docs: API keys, Service accounts
2) Store the API key (and API URL) in a Kubernetes Secret
Replace placeholders before running:
kubectl -n <your-namespace> create secret generic prefect-api \
--from-literal=PREFECT_API_KEY=<YOUR_API_KEY> \
--from-literal=PREFECT_API_URL="<https://api.prefect.cloud/api/accounts/<ACCOUNT_ID>/workspaces/<WORKSPACE_ID>>"
3) Create a Kubernetes work pool
prefect work-pool create "my-k8s-pool" --type kubernetes --paused
You can unpause later or start paused while you deploy the worker.
4) Apply minimal RBAC for the worker
This grants permission to create Jobs/Pods (and read Secrets/ConfigMaps) in a namespace.
apiVersion: v1
kind: ServiceAccount
metadata:
name: prefect-worker
namespace: <your-namespace>
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
name: prefect-worker-role
namespace: <your-namespace>
rules:
- apiGroups: [""]
resources: ["pods", "pods/log", "secrets", "configmaps"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
name: prefect-worker-rb
namespace: <your-namespace>
subjects:
- kind: ServiceAccount
name: prefect-worker
namespace: <your-namespace>
roleRef:
kind: Role
name: prefect-worker-role
apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
5) Deploy an in-cluster worker Deployment
Pin an image version you trust; 2-latest
works for quick starts.
apiVersion: apps/v1
kind: Deployment
metadata:
name: prefect-k8s-worker
namespace: <your-namespace>
spec:
replicas: 1
selector:
matchLabels:
app: prefect-k8s-worker
template:
metadata:
labels:
app: prefect-k8s-worker
spec:
serviceAccountName: prefect-worker
containers:
- name: worker
image: prefecthq/prefect:2-latest
imagePullPolicy: IfNotPresent
envFrom:
- secretRef:
name: prefect-api
args:
- prefect
- worker
- start
- -p
- my-k8s-pool
# Optional: set a name for the worker
# args: ["prefect","worker","start","-p","my-k8s-pool","--name","k8s-worker-1"]
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "1"
memory: "1Gi"
6) Unpause the pool and verify
- Unpause the pool in the UI or:
prefect work-pool resume "my-k8s-pool"
- Check the worker pod logs; it should connect to your workspace and start polling.
Optional: customize the job template (namespace, image, env, SA)
Fetch and edit the default base job template for your pool type:
prefect work-pool get-default-base-job-template --type kubernetes --file base-job-template.json
# edit base-job-template.json (e.g., set namespace, serviceAccountName, resources, env)
prefect work-pool create "my-k8s-pool" --type kubernetes --base-job-template base-job-template.json --overwrite
Docs for deeper reference
- Kubernetes worker: Kubernetes worker
- Kubernetes deployments overview: Deploying to Kubernetes
- API keys: API keysMarvin
09/04/2025, 11:22 PM