<@ULVA73B9P> i have deployed a prefect server and ...
# ask-marvin
y
@Marvin i have deployed a prefect server and worker on kubernetes. Now, i woudl like to define flows in a gitlab repo and have a continuous deployment workflow to deploy the defined flow to the prefect server / worker that i specify. how can this be done??
m
đź”§ Using:
explore_module_offerings
📊 Tools used: •
check_cli_command
(2x) •
display_callable_signature
(12x) •
explore_module_offerings
(5x) •
get_latest_prefect_release_notes
(2x) •
research_prefect_topic
(2x) •
review_common_3x_gotchas
(2x) •
search_prefect_3x_docs
(8x)
âś… thought for 287.5 seconds
đź”§ Using:
explore_module_offerings
📊 Tools used: •
display_callable_signature
(6x) •
explore_module_offerings
(9x) •
get_latest_prefect_release_notes
(3x) •
research_prefect_topic
(3x) •
review_common_3x_gotchas
(3x) •
search_prefect_3x_docs
(3x)
❌ Error: No module named 'prefect.Flow'
Short answer: yes—you can push code to GitLab, have CI create/update Prefect deployments that point to your GitLab repo as the source, and your Kubernetes worker will pick them up. It’s almost like the universe cares. Almost. I’ll assume Prefect 3.x. If you’re on 2.x, say so and I’ll adjust. What you need in place - A Kubernetes work pool and a worker already running in your cluster and connected to your Prefect server. - In GitLab CI/CD settings, set these masked variables: -
PREFECT_API_URL
pointing at your Prefect server’s API (e.g., http(s)://your-prefect-server/api) -
PREFECT_API_KEY
with a service account API key that can create deployments - Your worker must be able to pull your GitLab repo: - Best: use an SSH deploy key mounted into the worker pod and a
<mailto:git@gitlab.com|git@gitlab.com>:org/repo.git
URL. - Acceptable: use a read-only Personal Access Token and an HTTPS URL. Do not hardcode the token in the deployment; use masked env vars. Two common ways to wire up CI/CD Option A: Deploy via Python (explicit and flexible) - Add a tiny deploy script to your repo, e.g. `deploy.py`:
Copy code
from prefect import flow

# Optionally import flows here if you want to deploy from local source;
# but with from_source, you only need to reference the entrypoint.

if __name__ == "__main__":
    # One deployment example — repeat for each flow you want to deploy
    flow.from_source(
        source="<mailto:git@gitlab.com|git@gitlab.com>:org/repo.git",                # or <https://gitlab.com/org/repo.git>
        entrypoint="flows/etl.py:etl_flow",                  # path/from/repo/root.py:flow_function_name
    ).deploy(
        name="etl-prod",
        work_pool_name="k8s-pool",
        tags=["prod"],
        # parameters={"color": "blue"},                      # optional
        # description="ETL production deployment",           # optional
        # schedule=...,                                      # optional, can add later
    )
- Then add a GitLab CI job that runs this:
Copy code
image: python:3.11

stages: [deploy]

deploy-prefect:
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  before_script:
    - pip install "prefect>=3.0.0"
  script:
    - python deploy.py
Notes - This sets the deployment’s source to your GitLab repo and the entrypoint to the flow function. The worker will clone at run time. - Keep your Git credentials out of the code; use SSH deploy keys on the worker or masked variables. Option B: Use prefect.yaml + CLI (great for multiple deployments) - Create a
prefect.yaml
at your repo root:
Copy code
name: my-project
prefect-version: ">=3.0.0"
deployments:
  - name: etl-prod
    entrypoint: flows/etl.py:etl_flow
    work_pool:
      name: k8s-pool
      # Optional Kubernetes job variables (these come from your work pool template)
      # job_variables:
      #   image: <http://ghcr.io/your-org/your-image:tag|ghcr.io/your-org/your-image:tag>
      #   namespace: your-namespace
      #   service_account_name: your-sa
      #   env: [{"name": "EXAMPLE", "value": "1"}]
    # parameters:
    #   color: blue
    # schedule:  # optional
    #   cron: "0 * * * *"
- GitLab CI job:
Copy code
image: python:3.11

stages: [deploy]

deploy-prefect:
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  before_script:
    - pip install "prefect>=3.0.0"
  script:
    - prefect deploy --apply
- When run in CI within your repo,
prefect deploy
will detect the git remote and embed it as the deployment’s source so workers clone from GitLab at run time. Kubernetes worker and work pool tips - You likely already have these, but for completeness: - Create a Kubernetes work pool and start a worker in your cluster pointing to it. The worker’s pod needs RBAC to create jobs in your target namespace and network access to your Prefect server. - Use the work pool’s job template or job variables for things like container image, namespace, resources, service account, env, etc.
- Docs: Kubernetes work pools and workers Authentication to GitLab (so the worker can pull code) - SSH deploy key: mount the private key into the worker pod and use the
git@
URL. Add the public key as a deploy key on the repo. Safer, because you don’t embed tokens in deployment metadata. - HTTPS PAT: store a masked variable in your worker environment and rely on standard Git credential helpers in the worker image. Avoid baking tokens into the URL you pass to
from_source
or that ends up in
prefect.yaml
(they’re stored with the deployment). Good to know - Removed in Prefect 3.x (don’t use):
Deployment.build_from_flow()
and
prefect deployment build
. Prefer
flow.from_source(...).deploy(...)
or
prefect deploy
. - Your CI job only needs API access to your Prefect server to create/update deployments; the worker needs repo access to fetch code at run time. - Docs worth reading: - Store code in Git and deploy: https://docs-3.prefect.io/v3/how-to-guides/deployments/store-flow-code - Deploy via Python: https://docs-3.prefect.io/v3/how-to-guides/deployments/deploy-via-python - Kubernetes execution: https://docs-3.prefect.io/v3/how-to-guides/deployment_infra/kubernetes A few clarifying questions so I can tailor this precisely: - What’s the name of your Kubernetes work pool? - Is your GitLab repo private, and do you prefer SSH deploy keys or PATs? - Do you want to manage deployments via Python (Option A) or via
prefect.yaml
and CLI (Option B)? - Do you need schedules or parameters set as part of deployment?
y
@Marvin can you also give me a sample flow script for use with the gitlab deployment? and the project structure
m
âś… thought for 78.7 seconds
Absolutely. Here’s a simple, sane repo layout and a minimal flow you can drop into GitLab for Prefect 3.x deployments. It works cleanly with a Kubernetes work pool and a worker that clones your repo at run time. Project structure
Copy code
your-repo/
├─ flows/
│  └─ etl.py
├─ prefect.yaml
├─ requirements.txt
└─ .gitlab-ci.yml        # optional but handy for CI/CD
Sample flow script (flows/etl.py)
Copy code
from prefect import flow, task, get_run_logger

@task
def extract():
    return ["alpha", "beta", "gamma", "delta"]

@task
def transform(items, uppercase: bool = True):
    return [s.upper() for s in items] if uppercase else items

@task
def load(items):
    logger = get_run_logger()
    for item in items:
        <http://logger.info|logger.info>(f"Loaded item: {item}")
    return len(items)

@flow(name="etl-flow")
def etl_flow(limit: int = 3, uppercase: bool = True):
    logger = get_run_logger()
    data = extract()
    data = transform(data, uppercase=uppercase)
    data = data[:limit]
    count = load(data)
    <http://logger.info|logger.info>(f"ETL finished, loaded {count} items.")

if __name__ == "__main__":
    # Allows local testing without Prefect deployment
    etl_flow()
prefect.yaml (CLI-driven deployments; recommended for CI/CD)
Copy code
name: gitlab-prefect-project
prefect-version: ">=3.0.0"

deployments:
  - name: etl-prod
    entrypoint: flows/etl.py:etl_flow
    parameters:
      limit: 2
      uppercase: true
    tags: ["prod"]
    work_pool:
      name: k8s-pool
      # Optional: job_variables for Kubernetes template, e.g. image, namespace, resources, env, etc.
      # job_variables:
      #   image: "<http://ghcr.io/your-org/your-image:latest|ghcr.io/your-org/your-image:latest>"
      #   namespace: "your-namespace"
      #   env:
      #     - name: EXAMPLE
      #       value: "1"
    # Optional schedule example:
    # schedule:
    #   cron: "0 * * * *"
    #   timezone: "UTC"
requirements.txt
Copy code
prefect>=3.0.0
Optional: GitLab CI job to continuously deploy
Copy code
image: python:3.11

stages: [deploy]

deploy-prefect:
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  before_script:
    - pip install -r requirements.txt
  script:
    # Creates/updates deployments declared in prefect.yaml
    - prefect deploy --apply
Notes that will save you a headache later - Ensure these GitLab masked variables exist for the CI job: - PREFECT_API_URL (e.g., http(s)://your-prefect-server/api) - PREFECT_API_KEY (service account with deployment permissions) - Your Kubernetes worker must be able to clone the repo at run time: - Prefer SSH deploy keys and use a
<mailto:git@gitlab.com|git@gitlab.com>:org/repo.git
remote. - If you must use a PAT over HTTPS, keep it out of source; rely on masked env vars and your worker image’s git credentials config. - Don’t use Prefect 2.x deployment commands. In 3.x, the CLI you want is
prefect deploy
. The 2.x
prefect deployment build
has been retired to the great beyond. If you prefer Python-driven deployment instead of prefect.yaml Add a tiny deploy script (deploy.py) and run it in CI:
Copy code
from prefect import flow

if __name__ == "__main__":
    # This tells the deployment where to fetch code from at run time
    flow.from_source(
        source="<mailto:git@gitlab.com|git@gitlab.com>:org/repo.git",      # or <https://gitlab.com/org/repo.git>
        entrypoint="flows/etl.py:etl_flow",
        # ref="main",                               # optional: pin to branch or tag/commit
    ).deploy(
        name="etl-prod",
        work_pool_name="k8s-pool",
        parameters={"limit": 2, "uppercase": True},
        tags=["prod"],
        # schedule=...,                            # optional
    )
Then in CI:
Copy code
image: python:3.11
stages: [deploy]

deploy-prefect:
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  before_script:
    - pip install -r requirements.txt
  script:
    - python deploy.py
Docs, because of course there are docs:
- Store flow code in Git and deploy: Store flow code in Git - Deploy with prefect.yaml or Python: Deploy via Python / CLI - Kubernetes work pools/workers: Kubernetes execution If you share your work pool name and whether you’re using SSH or PAT for GitLab, I can tailor the job variables and auth bits so your worker doesn’t sulk at runtime.
y
@Marvin must the image under job variable have prefect installed?
m
âś… thought for 50.0 seconds
Short answer: yes. The image you set in the work pool’s job variables is the container that actually runs your flow. It needs Prefect installed (plus your flow’s dependencies). The worker pod is separate; it also needs Prefect, but that’s a different image. What the flow-run image should include - Python and prefect (e.g., pip install prefect>=3) - Your flow dependencies (pandas, requests, etc.) - Git client if your deployment pulls code from GitLab at run time (recommended to install git and openssh-client if using SSH) - Any runtime tools (CA certs, system libs your deps need) Minimal Dockerfile example
Copy code
FROM python:3.11-slim

# System tools if you pull code from Git (HTTPS or SSH)
RUN apt-get update && apt-get install -y --no-install-recommends git openssh-client \
    && rm -rf /var/lib/apt/lists/*

# Python deps
RUN pip install --no-cache-dir "prefect>=3.0.0"

# Your app deps
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Optional: set a working directory
WORKDIR /opt/prefect
Set the image in your deployment (prefect.yaml)
Copy code
work_pool:
  name: k8s-pool
  job_variables:
    image: <http://registry.gitlab.com/your-org/your-image:latest|registry.gitlab.com/your-org/your-image:latest>
    # optionally namespace, service_account_name, env, resources, etc.
A couple of gotchas to avoid - If your deployment uses a GitLab repo as the source, the flow-run image needs to be able to clone it. That means: - Install git (and openssh-client if using SSH) - Provide credentials via Kubernetes secrets/env/volumes. Don’t bake tokens into the image. - You do not need to bake PREFECT_API_URL/API_KEY into the image; the worker injects the necessary env for each run. - If you don’t want to build a custom image, you can start from a Python base and pip install prefect and your deps in the job variables via an init step, but building an image is far more reliable. If you share whether you’re using SSH keys or PAT for GitLab, I can sketch the exact env/volume mounts for the work pool job template so your runs clone cleanly.
y
@Marvin does it pull code from gitlab by default?
m
âś… thought for 43.0 seconds
Short answer: not universally “by default.” It pulls from GitLab at run time only if your deployment includes a Git source. Otherwise, it expects the code to already be in the container image. Here’s how it behaves: - If you create the deployment with flow.from_source(...).deploy(...): the deployment records your GitLab repo and entrypoint, and the worker clones that repo at run time. - If you run prefect deploy from inside your Git repo (with a prefect.yaml): the CLI auto-detects the repo and sets it as the deployment’s source; the worker clones at run time. - If your deployment has no source (e.g., you built an image that already contains your code and pointed the entrypoint at local files): no Git pull happens; Prefect loads the flow from the container filesystem. Practical guidance: - Want it to pull from GitLab? Use either: - Python API:
Copy code
from prefect import flow

    flow.from_source(
        source="<mailto:git@gitlab.com|git@gitlab.com>:org/repo.git",            # or <https://gitlab.com/org/repo.git>
        entrypoint="flows/etl.py:etl_flow",
        # ref="main",                                     # optional: pin branch/tag/commit
    ).deploy(
        name="etl-prod",
        work_pool_name="k8s-pool",
    )
- CLI from your repo with a prefect.yaml:
Copy code
prefect deploy --apply
Run that in the repo root; the CLI will record your Git remote as the source. - Want to avoid pulling from GitLab? Bake your flow code into the image and don’t set a source. Your entrypoint should point to files present in the image, and the image must include Prefect and your deps. Checks and gotchas: - In the Prefect UI, open the deployment: if you see a “Source” pointing to your GitLab URL and an entrypoint, it will clone at run time. - If pulling from GitLab, your flow-run image needs git (and openssh-client if using SSH) plus credentials via K8s secrets/env. If not pulling, you don’t need git in the image. - Docs, if you enjoy light reading: - Store flow code in Git: Store flow code in Git - Deploy via Python/CLI: Deploy via Python / CLI - Kubernetes workers: Kubernetes execution If you tell me how you created the deployment (Python vs CLI) and whether you want image-baked or git-cloned code, I’ll give you the exact setup. I promise only a minimal amount of existential dread.
y
@Marvin how do i pass the deploykey onto the worker?
m
âś… thought for 86.5 seconds
Short version: you don’t “pass it to the worker” so much as mount it into the flow-run pod created by the worker. The worker schedules Jobs; the pod that runs your flow needs the deploy key to git clone your repo at runtime. Two solid ways to do it (SSH deploy key recommended) A) Mount an SSH deploy key secret into the flow-run pod 1) Create a Kubernetes secret with your private key (and known_hosts for GitLab):
Copy code
# Private key file should be unencrypted and read-only, e.g. id_ed25519
kubectl create secret generic git-deploy-key \
  --from-file=id_ed25519=./id_ed25519 \
  --from-file=known_hosts=./known_hosts \
  -n your-namespace
- Generate known_hosts with:
Copy code
ssh-keyscan <http://gitlab.com|gitlab.com> > known_hosts
Keep StrictHostKeyChecking on; you’ll sleep better. 2) Update your Kubernetes work pool’s job template to mount the secret and set GIT_SSH_COMMAND. - In the Prefect UI, open your k8s work pool and edit the job template to add: - A volume for the secret - A volumeMount on the main container (the one that runs your flow) - An env var to tell git which key to use and to honor known_hosts Example additions you want in the pod spec (structure may vary depending on your template):
Copy code
spec:
  volumes:
    - name: ssh-key
      secret:
        secretName: git-deploy-key
        defaultMode: 0400
  containers:
    - name: prefect-job
      volumeMounts:
        - name: ssh-key
          mountPath: /root/.ssh
          readOnly: true
      env:
        - name: GIT_SSH_COMMAND
          value: "ssh -i /root/.ssh/id_ed25519 -o UserKnownHostsFile=/root/.ssh/known_hosts -o StrictHostKeyChecking=yes"
Notes: - If your image runs as a non-root user, mount at that user’s home (e.g., /home/app/.ssh) and ensure file permissions are acceptable for that UID. - Many Prefect k8s templates expose a “pod” override or similar under job_variables. If your template doesn’t, duplicate/customize the template to include volumes/volumeMounts. 3) In your deployment, use the SSH URL:
Copy code
source="<mailto:git@gitlab.com|git@gitlab.com>:org/repo.git"
Now every flow run will clone using the mounted key. B) Alternative: HTTPS with a PAT (simpler wiring, less ideal) - Store a read-only PAT in a Kubernetes secret or use a Kubernetes Secret as an env var. - Configure git credentials in the container (e.g., via env and a small init step) so
<https://gitlab.com/org/repo.git>
can be cloned. Avoid embedding the token in the URL in your deployment; let the runtime environment provide it. - You still need git installed in the flow-run image. Image requirements (either approach) - The flow-run image must include: - Python and
prefect
(and your deps) - git (and openssh-client if using SSH) - You don’t need to bake `PREFECT_API_URL`/`PREFECT_API_KEY` into the image; Prefect injects them at runtime. If you want a concrete template snippet - Tell me: - Your work pool name - Namespace - Whether your image runs as root or a specific user - SSH vs PAT And I’ll hand you the exact job template block to paste into the pool. Useful docs, because of course: - Store flow code in Git: Store flow code in Git - Kubernetes workers/pools: Kubernetes execution
y
@Marvin does that mean i will have to modify the jobtemplate within the prefect worker helm chart values for this?
m
âś… thought for 72.8 seconds
Short answer: no. You don’t need to change the worker Helm chart. The Helm chart configures the worker pod. The deploy key needs to be mounted into the flow-run pods, which are controlled by your Kubernetes work pool’s job template (and per-deployment job variables), not the worker chart. What to do instead - Put the SSH key in a Kubernetes Secret. - Update the Kubernetes work pool’s job template (in the Prefect UI or via API) to: - Mount that secret as a volume - Mount it into the main container - Set GIT_SSH_COMMAND so git uses your key and known_hosts - Optionally, if you only want this for specific deployments, set the same things in the deployment’s job_variables (prefect.yaml) instead of the pool template. Step-by-step 1) Create a secret with your private key and known_hosts
Copy code
# id_ed25519 should be your unencrypted private key
ssh-keyscan <http://gitlab.com|gitlab.com> > known_hosts

kubectl -n your-namespace create secret generic git-deploy-key \
  --from-file=id_ed25519=./id_ed25519 \
  --from-file=known_hosts=./known_hosts
2) In your Kubernetes work pool, edit the job template (or set per-deployment job_variables) to mount it Example to put in prefect.yaml under a deployment’s work_pool.job_variables (same structure applies in the pool template if you want it globally):
Copy code
work_pool:
  name: k8s-pool
  job_variables:
    image: <http://registry.gitlab.com/your-org/your-image:latest|registry.gitlab.com/your-org/your-image:latest>
    env:
      - name: GIT_SSH_COMMAND
        value: "ssh -i /root/.ssh/id_ed25519 -o UserKnownHostsFile=/root/.ssh/known_hosts -o StrictHostKeyChecking=yes"
    pod:
      volumes:
        - name: ssh-key
          secret:
            secretName: git-deploy-key
            defaultMode: 0600
      containers:
        - name: prefect-job
          volumeMounts:
            - name: ssh-key
              mountPath: /root/.ssh
              readOnly: true
      # Optional safety if your cluster ignores defaultMode on Secrets
      initContainers:
        - name: fix-perms
          image: busybox:1.36
          command: ["sh", "-c", "chmod 600 /root/.ssh/id_ed25519; chmod 644 /root/.ssh/known_hosts"]
          volumeMounts:
            - name: ssh-key
              mountPath: /root/.ssh
3) Use the SSH clone URL in your deployment’s source
Copy code
source="<mailto:git@gitlab.com|git@gitlab.com>:org/repo.git"
Notes - The flow-run image must include git and openssh-client, plus prefect and your deps. - If your container runs as a non-root user, adjust mountPath to that user’s home and ensure perms match that UID. - Configure it at the work pool level if all flows need it; otherwise use per-deployment job_variables. Docs to cross-check me (and keep your future self sane) - Store flow code in Git - Kubernetes work pools and workers
y
@Marvin can i use PAT instead of deploy key?
m
âś… thought for 38.8 seconds
Yes. You can use a GitLab Personal Access Token (or better, a Project/Deploy Token) with HTTPS instead of an SSH deploy key. The token just needs to be available to the flow-run pod so git can authenticate when Prefect clones your repo at runtime. Two solid recipes A) Store a PAT in .git-credentials (simple and reliable) 1) Create the files locally:
Copy code
# For GitLab, username `oauth2` + PAT works well for HTTPS
echo "<https://oauth2:${GITLAB_PAT}@gitlab.com>" > .git-credentials

# Tell git to use the credentials file
printf "[credential]\n\thelper = store\n" > .gitconfig
2) Create a Kubernetes Secret:
Copy code
kubectl -n your-namespace create secret generic gitlab-https-creds \
  --from-file=.git-credentials \
  --from-file=.gitconfig
3) Mount them into the flow-run container via your Kubernetes work pool job template (or per-deployment job_variables in prefect.yaml). Example:
Copy code
work_pool:
  name: k8s-pool
  job_variables:
    image: <http://registry.gitlab.com/your-org/your-image:latest|registry.gitlab.com/your-org/your-image:latest>
    pod:
      volumes:
        - name: git-creds
          secret:
            secretName: gitlab-https-creds
            defaultMode: 0600
      containers:
        - name: prefect-job
          volumeMounts:
            - name: git-creds
              mountPath: /root/.git-credentials
              subPath: .git-credentials
              readOnly: true
            - name: git-creds
              mountPath: /root/.gitconfig
              subPath: .gitconfig
              readOnly: true
Notes: - Adjust paths if your image runs as a non-root user (e.g., /home/app). - The flow-run image must have git installed. B) Use a GitLab Deploy Token (scoped, recommended over user PATs) - Create a Deploy Token with read_repository scope. - Same mechanics as above, except your .git-credentials line will look like:
Copy code
https://<deploy_token_username>:<deploy_token>@gitlab.com
- This avoids tying access to a human account and is easier to rotate. Things not to do (unless you enjoy incident reports) - Don’t bake the token into the deployment source URL (e.g., https://oauth2:TOKEN@gitlab.com/org/repo.git). Prefect will store deployment metadata; you don’t want your token in there. - Don’t put the token in your image. Use Kubernetes Secrets and mount them at runtime. Quick checklist - Worker pool job template mounts the secret and your image has git + prefect + your deps. - Your deployment’s source points to the HTTPS URL (no credentials in it), e.g.:
Copy code
source="<https://gitlab.com/org/repo.git>"
  entrypoint="flows/etl.py:etl_flow"
- Token rotation = update the Kubernetes Secret. No redeploy needed unless you hardcoded it somewhere (please don’t). Docs for further comfort reading - Store flow code in Git - Kubernetes work pools and workers If you share your namespace, pool name, and whether your container runs as root or a specific UID, I’ll give you a paste-ready job_variables block.
y
@Marvin PREFECT_API_KEY does not apply to self-hosted servers right?
m
âś… thought for 534.5 seconds
âś… thought for 232.0 seconds
Short answer: correct.
PREFECT_API_KEY
is only for Prefect Cloud. A self-hosted Prefect Server doesn’t use it. For Server, point your client/agents at your API and (optionally) UI: - Set API URL:
Copy code
prefect config set PREFECT_API_URL=http://<host>:4200/api
- Optional UI URL (for links in logs, etc.):
Copy code
prefect config set PREFECT_UI_URL=http://<host>:4200
- If you’re putting the API behind your own auth (reverse proxy), use:
Copy code
prefect config set PREFECT_API_AUTH_STRING="Bearer <your-token>"   # or "Basic <base64-creds>"
If you previously used Cloud, it’s safest to clear the key so nothing tries to “helpfully” authenticate to the wrong place:
Copy code
prefect config unset PREFECT_API_KEY
If you need TLS knobs, there are settings for skipping verification or pointing at a cert file as well.
PREFECT_API_KEY is the secret token Prefect uses to authenticate you (or your worker) to Prefect Cloud. How to create and use it (Prefect 3.x): 1) Create an API key - In Prefect Cloud, open your user menu (top-right) → API Keys → Create key. - Copy it once; you won’t see it again. Yes, it’s shy. 2) Use it with the CLI - One-shot login (sets up a profile and workspace):
Copy code
prefect cloud login -k <YOUR_API_KEY> -w <account_handle>/<workspace_handle>
- Change the current workspace later:
Copy code
prefect cloud workspace set -w <account_handle>/<workspace_handle>
- Logout (clears API URL/key in the current profile):
Copy code
prefect cloud logout
3) Set it via environment variable - Unix/macOS:
Copy code
export PREFECT_API_KEY="<YOUR_API_KEY>"
# Optional if you are not using `prefect cloud login`:
# export PREFECT_API_URL="<workspace API URL>"
- Windows (PowerShell):
Copy code
$env:PREFECT_API_KEY="<YOUR_API_KEY>"
# Optional:
# $env:PREFECT_API_URL="<workspace API URL>"
Tip: You can persist it in your Prefect profile instead of env vars:
Copy code
prefect config set PREFECT_API_KEY=<YOUR_API_KEY>
4) Programmatic use - Most code should rely on your configured profile/env vars:
Copy code
import asyncio
from prefect.client.orchestration import get_client

async def main():
    async with get_client() as client:
        # you're authenticated via PREFECT_API_KEY/profile here
        ...
asyncio.run(main())
- Or pass the key directly (and the API URL if needed):
Copy code
import asyncio
from prefect.client.orchestration import PrefectClient

async def main():
    async with PrefectClient(
        api="<your workspace API URL>",
        api_key="<YOUR_API_KEY>",
    ) as client:
        ...
asyncio.run(main())
Notes - Local Prefect Server (self-hosted) doesn’t require an API key. - Workers/runners in containers/CI typically set
PREFECT_API_KEY
(and often
PREFECT_API_URL
) via environment variables. - You can view current settings and their sources:
Copy code
prefect config view --show-sources
Docs - Settings and profiles (includes
PREFECT_API_KEY
): Prefect Settings and Profiles - CLI reference: Prefect CLI - Cloud UI (create API keys): Prefect Cloud If you want, tell me your OS and whether you’re using a worker/runner, and I’ll give you the exact setup commands.
y
@Marvin where do i define the source? source="https://gitlab.com/org/repo.git"
m
❌ Error: No module named 'prefect.storage'
❌ Error: No module named 'prefect.storage'
y
@Marvin i managed to deploy a flow onto the prefect instance but the flow fails with not being able to set working directory. where is this specified and how can i work around it?
m
❌ Error: No module named 'prefect.workers.docker'
❌ Error: No module named 'prefect.runners'