Yu Shao Pang
08/11/2025, 1:26 PMMarvin
08/11/2025, 1:26 PMexplore_module_offerings
📊 Tools used:
• check_cli_command
(2x)
• display_callable_signature
(12x)
• explore_module_offerings
(5x)
• get_latest_prefect_release_notes
(2x)
• research_prefect_topic
(2x)
• review_common_3x_gotchas
(2x)
• search_prefect_3x_docs
(8x)Marvin
08/11/2025, 1:28 PMMarvin
08/11/2025, 1:31 PMexplore_module_offerings
📊 Tools used:
• display_callable_signature
(6x)
• explore_module_offerings
(9x)
• get_latest_prefect_release_notes
(3x)
• research_prefect_topic
(3x)
• review_common_3x_gotchas
(3x)
• search_prefect_3x_docs
(3x)Marvin
08/11/2025, 1:32 PMMarvin
08/11/2025, 1:32 PMPREFECT_API_URL
pointing at your Prefect server’s API (e.g., http(s)://your-prefect-server/api)
- PREFECT_API_KEY
with a service account API key that can create deployments
- Your worker must be able to pull your GitLab repo:
- Best: use an SSH deploy key mounted into the worker pod and a <mailto:git@gitlab.com|git@gitlab.com>:org/repo.git
URL.
- Acceptable: use a read-only Personal Access Token and an HTTPS URL. Do not hardcode the token in the deployment; use masked env vars.
Two common ways to wire up CI/CD
Option A: Deploy via Python (explicit and flexible)
- Add a tiny deploy script to your repo, e.g. `deploy.py`:
from prefect import flow
# Optionally import flows here if you want to deploy from local source;
# but with from_source, you only need to reference the entrypoint.
if __name__ == "__main__":
# One deployment example — repeat for each flow you want to deploy
flow.from_source(
source="<mailto:git@gitlab.com|git@gitlab.com>:org/repo.git", # or <https://gitlab.com/org/repo.git>
entrypoint="flows/etl.py:etl_flow", # path/from/repo/root.py:flow_function_name
).deploy(
name="etl-prod",
work_pool_name="k8s-pool",
tags=["prod"],
# parameters={"color": "blue"}, # optional
# description="ETL production deployment", # optional
# schedule=..., # optional, can add later
)
- Then add a GitLab CI job that runs this:
image: python:3.11
stages: [deploy]
deploy-prefect:
stage: deploy
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
before_script:
- pip install "prefect>=3.0.0"
script:
- python deploy.py
Notes
- This sets the deployment’s source to your GitLab repo and the entrypoint to the flow function. The worker will clone at run time.
- Keep your Git credentials out of the code; use SSH deploy keys on the worker or masked variables.
Option B: Use prefect.yaml + CLI (great for multiple deployments)
- Create a prefect.yaml
at your repo root:
name: my-project
prefect-version: ">=3.0.0"
deployments:
- name: etl-prod
entrypoint: flows/etl.py:etl_flow
work_pool:
name: k8s-pool
# Optional Kubernetes job variables (these come from your work pool template)
# job_variables:
# image: <http://ghcr.io/your-org/your-image:tag|ghcr.io/your-org/your-image:tag>
# namespace: your-namespace
# service_account_name: your-sa
# env: [{"name": "EXAMPLE", "value": "1"}]
# parameters:
# color: blue
# schedule: # optional
# cron: "0 * * * *"
- GitLab CI job:
image: python:3.11
stages: [deploy]
deploy-prefect:
stage: deploy
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
before_script:
- pip install "prefect>=3.0.0"
script:
- prefect deploy --apply
- When run in CI within your repo, prefect deploy
will detect the git remote and embed it as the deployment’s source so workers clone from GitLab at run time.
Kubernetes worker and work pool tips
- You likely already have these, but for completeness:
- Create a Kubernetes work pool and start a worker in your cluster pointing to it. The worker’s pod needs RBAC to create jobs in your target namespace and network access to your Prefect server.
- Use the work pool’s job template or job variables for things like container image, namespace, resources, service account, env, etc.Marvin
08/11/2025, 1:32 PMgit@
URL. Add the public key as a deploy key on the repo. Safer, because you don’t embed tokens in deployment metadata.
- HTTPS PAT: store a masked variable in your worker environment and rely on standard Git credential helpers in the worker image. Avoid baking tokens into the URL you pass to from_source
or that ends up in prefect.yaml
(they’re stored with the deployment).
Good to know
- Removed in Prefect 3.x (don’t use): Deployment.build_from_flow()
and prefect deployment build
. Prefer flow.from_source(...).deploy(...)
or prefect deploy
.
- Your CI job only needs API access to your Prefect server to create/update deployments; the worker needs repo access to fetch code at run time.
- Docs worth reading:
- Store code in Git and deploy: https://docs-3.prefect.io/v3/how-to-guides/deployments/store-flow-code
- Deploy via Python: https://docs-3.prefect.io/v3/how-to-guides/deployments/deploy-via-python
- Kubernetes execution: https://docs-3.prefect.io/v3/how-to-guides/deployment_infra/kubernetes
A few clarifying questions so I can tailor this precisely:
- What’s the name of your Kubernetes work pool?
- Is your GitLab repo private, and do you prefer SSH deploy keys or PATs?
- Do you want to manage deployments via Python (Option A) or via prefect.yaml
and CLI (Option B)?
- Do you need schedules or parameters set as part of deployment?Yu Shao Pang
08/11/2025, 1:39 PMMarvin
08/11/2025, 1:39 PMMarvin
08/11/2025, 1:40 PMyour-repo/
├─ flows/
│ └─ etl.py
├─ prefect.yaml
├─ requirements.txt
└─ .gitlab-ci.yml # optional but handy for CI/CD
Sample flow script (flows/etl.py)
from prefect import flow, task, get_run_logger
@task
def extract():
return ["alpha", "beta", "gamma", "delta"]
@task
def transform(items, uppercase: bool = True):
return [s.upper() for s in items] if uppercase else items
@task
def load(items):
logger = get_run_logger()
for item in items:
<http://logger.info|logger.info>(f"Loaded item: {item}")
return len(items)
@flow(name="etl-flow")
def etl_flow(limit: int = 3, uppercase: bool = True):
logger = get_run_logger()
data = extract()
data = transform(data, uppercase=uppercase)
data = data[:limit]
count = load(data)
<http://logger.info|logger.info>(f"ETL finished, loaded {count} items.")
if __name__ == "__main__":
# Allows local testing without Prefect deployment
etl_flow()
prefect.yaml (CLI-driven deployments; recommended for CI/CD)
name: gitlab-prefect-project
prefect-version: ">=3.0.0"
deployments:
- name: etl-prod
entrypoint: flows/etl.py:etl_flow
parameters:
limit: 2
uppercase: true
tags: ["prod"]
work_pool:
name: k8s-pool
# Optional: job_variables for Kubernetes template, e.g. image, namespace, resources, env, etc.
# job_variables:
# image: "<http://ghcr.io/your-org/your-image:latest|ghcr.io/your-org/your-image:latest>"
# namespace: "your-namespace"
# env:
# - name: EXAMPLE
# value: "1"
# Optional schedule example:
# schedule:
# cron: "0 * * * *"
# timezone: "UTC"
requirements.txt
prefect>=3.0.0
Optional: GitLab CI job to continuously deploy
image: python:3.11
stages: [deploy]
deploy-prefect:
stage: deploy
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
before_script:
- pip install -r requirements.txt
script:
# Creates/updates deployments declared in prefect.yaml
- prefect deploy --apply
Notes that will save you a headache later
- Ensure these GitLab masked variables exist for the CI job:
- PREFECT_API_URL (e.g., http(s)://your-prefect-server/api)
- PREFECT_API_KEY (service account with deployment permissions)
- Your Kubernetes worker must be able to clone the repo at run time:
- Prefer SSH deploy keys and use a <mailto:git@gitlab.com|git@gitlab.com>:org/repo.git
remote.
- If you must use a PAT over HTTPS, keep it out of source; rely on masked env vars and your worker image’s git credentials config.
- Don’t use Prefect 2.x deployment commands. In 3.x, the CLI you want is prefect deploy
. The 2.x prefect deployment build
has been retired to the great beyond.
If you prefer Python-driven deployment instead of prefect.yaml
Add a tiny deploy script (deploy.py) and run it in CI:
from prefect import flow
if __name__ == "__main__":
# This tells the deployment where to fetch code from at run time
flow.from_source(
source="<mailto:git@gitlab.com|git@gitlab.com>:org/repo.git", # or <https://gitlab.com/org/repo.git>
entrypoint="flows/etl.py:etl_flow",
# ref="main", # optional: pin to branch or tag/commit
).deploy(
name="etl-prod",
work_pool_name="k8s-pool",
parameters={"limit": 2, "uppercase": True},
tags=["prod"],
# schedule=..., # optional
)
Then in CI:
image: python:3.11
stages: [deploy]
deploy-prefect:
stage: deploy
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
before_script:
- pip install -r requirements.txt
script:
- python deploy.py
Docs, because of course there are docs:Marvin
08/11/2025, 1:40 PMYu Shao Pang
08/11/2025, 3:53 PMMarvin
08/11/2025, 3:53 PMMarvin
08/11/2025, 3:54 PMFROM python:3.11-slim
# System tools if you pull code from Git (HTTPS or SSH)
RUN apt-get update && apt-get install -y --no-install-recommends git openssh-client \
&& rm -rf /var/lib/apt/lists/*
# Python deps
RUN pip install --no-cache-dir "prefect>=3.0.0"
# Your app deps
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Optional: set a working directory
WORKDIR /opt/prefect
Set the image in your deployment (prefect.yaml)
work_pool:
name: k8s-pool
job_variables:
image: <http://registry.gitlab.com/your-org/your-image:latest|registry.gitlab.com/your-org/your-image:latest>
# optionally namespace, service_account_name, env, resources, etc.
A couple of gotchas to avoid
- If your deployment uses a GitLab repo as the source, the flow-run image needs to be able to clone it. That means:
- Install git (and openssh-client if using SSH)
- Provide credentials via Kubernetes secrets/env/volumes. Don’t bake tokens into the image.
- You do not need to bake PREFECT_API_URL/API_KEY into the image; the worker injects the necessary env for each run.
- If you don’t want to build a custom image, you can start from a Python base and pip install prefect and your deps in the job variables via an init step, but building an image is far more reliable.
If you share whether you’re using SSH keys or PAT for GitLab, I can sketch the exact env/volume mounts for the work pool job template so your runs clone cleanly.Yu Shao Pang
08/11/2025, 3:55 PMMarvin
08/11/2025, 3:55 PMMarvin
08/11/2025, 3:56 PMfrom prefect import flow
flow.from_source(
source="<mailto:git@gitlab.com|git@gitlab.com>:org/repo.git", # or <https://gitlab.com/org/repo.git>
entrypoint="flows/etl.py:etl_flow",
# ref="main", # optional: pin branch/tag/commit
).deploy(
name="etl-prod",
work_pool_name="k8s-pool",
)
- CLI from your repo with a prefect.yaml:
prefect deploy --apply
Run that in the repo root; the CLI will record your Git remote as the source.
- Want to avoid pulling from GitLab? Bake your flow code into the image and don’t set a source. Your entrypoint should point to files present in the image, and the image must include Prefect and your deps.
Checks and gotchas:
- In the Prefect UI, open the deployment: if you see a “Source” pointing to your GitLab URL and an entrypoint, it will clone at run time.
- If pulling from GitLab, your flow-run image needs git (and openssh-client if using SSH) plus credentials via K8s secrets/env. If not pulling, you don’t need git in the image.
- Docs, if you enjoy light reading:
- Store flow code in Git: Store flow code in Git
- Deploy via Python/CLI: Deploy via Python / CLI
- Kubernetes workers: Kubernetes execution
If you tell me how you created the deployment (Python vs CLI) and whether you want image-baked or git-cloned code, I’ll give you the exact setup. I promise only a minimal amount of existential dread.Yu Shao Pang
08/11/2025, 3:57 PMMarvin
08/11/2025, 3:58 PMMarvin
08/11/2025, 3:59 PM# Private key file should be unencrypted and read-only, e.g. id_ed25519
kubectl create secret generic git-deploy-key \
--from-file=id_ed25519=./id_ed25519 \
--from-file=known_hosts=./known_hosts \
-n your-namespace
- Generate known_hosts with:
ssh-keyscan <http://gitlab.com|gitlab.com> > known_hosts
Keep StrictHostKeyChecking on; you’ll sleep better.
2) Update your Kubernetes work pool’s job template to mount the secret and set GIT_SSH_COMMAND.
- In the Prefect UI, open your k8s work pool and edit the job template to add:
- A volume for the secret
- A volumeMount on the main container (the one that runs your flow)
- An env var to tell git which key to use and to honor known_hosts
Example additions you want in the pod spec (structure may vary depending on your template):
spec:
volumes:
- name: ssh-key
secret:
secretName: git-deploy-key
defaultMode: 0400
containers:
- name: prefect-job
volumeMounts:
- name: ssh-key
mountPath: /root/.ssh
readOnly: true
env:
- name: GIT_SSH_COMMAND
value: "ssh -i /root/.ssh/id_ed25519 -o UserKnownHostsFile=/root/.ssh/known_hosts -o StrictHostKeyChecking=yes"
Notes:
- If your image runs as a non-root user, mount at that user’s home (e.g., /home/app/.ssh) and ensure file permissions are acceptable for that UID.
- Many Prefect k8s templates expose a “pod” override or similar under job_variables. If your template doesn’t, duplicate/customize the template to include volumes/volumeMounts.
3) In your deployment, use the SSH URL:
source="<mailto:git@gitlab.com|git@gitlab.com>:org/repo.git"
Now every flow run will clone using the mounted key.
B) Alternative: HTTPS with a PAT (simpler wiring, less ideal)
- Store a read-only PAT in a Kubernetes secret or use a Kubernetes Secret as an env var.
- Configure git credentials in the container (e.g., via env and a small init step) so <https://gitlab.com/org/repo.git>
can be cloned. Avoid embedding the token in the URL in your deployment; let the runtime environment provide it.
- You still need git installed in the flow-run image.
Image requirements (either approach)
- The flow-run image must include:
- Python and prefect
(and your deps)
- git (and openssh-client if using SSH)
- You don’t need to bake `PREFECT_API_URL`/`PREFECT_API_KEY` into the image; Prefect injects them at runtime.
If you want a concrete template snippet
- Tell me:
- Your work pool name
- Namespace
- Whether your image runs as root or a specific user
- SSH vs PAT
And I’ll hand you the exact job template block to paste into the pool.
Useful docs, because of course:
- Store flow code in Git: Store flow code in Git
- Kubernetes workers/pools: Kubernetes executionYu Shao Pang
08/11/2025, 4:09 PMMarvin
08/11/2025, 4:09 PMMarvin
08/11/2025, 4:10 PM# id_ed25519 should be your unencrypted private key
ssh-keyscan <http://gitlab.com|gitlab.com> > known_hosts
kubectl -n your-namespace create secret generic git-deploy-key \
--from-file=id_ed25519=./id_ed25519 \
--from-file=known_hosts=./known_hosts
2) In your Kubernetes work pool, edit the job template (or set per-deployment job_variables) to mount it
Example to put in prefect.yaml under a deployment’s work_pool.job_variables (same structure applies in the pool template if you want it globally):
work_pool:
name: k8s-pool
job_variables:
image: <http://registry.gitlab.com/your-org/your-image:latest|registry.gitlab.com/your-org/your-image:latest>
env:
- name: GIT_SSH_COMMAND
value: "ssh -i /root/.ssh/id_ed25519 -o UserKnownHostsFile=/root/.ssh/known_hosts -o StrictHostKeyChecking=yes"
pod:
volumes:
- name: ssh-key
secret:
secretName: git-deploy-key
defaultMode: 0600
containers:
- name: prefect-job
volumeMounts:
- name: ssh-key
mountPath: /root/.ssh
readOnly: true
# Optional safety if your cluster ignores defaultMode on Secrets
initContainers:
- name: fix-perms
image: busybox:1.36
command: ["sh", "-c", "chmod 600 /root/.ssh/id_ed25519; chmod 644 /root/.ssh/known_hosts"]
volumeMounts:
- name: ssh-key
mountPath: /root/.ssh
3) Use the SSH clone URL in your deployment’s source
source="<mailto:git@gitlab.com|git@gitlab.com>:org/repo.git"
Notes
- The flow-run image must include git and openssh-client, plus prefect and your deps.
- If your container runs as a non-root user, adjust mountPath to that user’s home and ensure perms match that UID.
- Configure it at the work pool level if all flows need it; otherwise use per-deployment job_variables.
Docs to cross-check me (and keep your future self sane)
- Store flow code in Git
- Kubernetes work pools and workersYu Shao Pang
08/11/2025, 4:21 PMMarvin
08/11/2025, 4:21 PMMarvin
08/11/2025, 4:21 PM# For GitLab, username `oauth2` + PAT works well for HTTPS
echo "<https://oauth2:${GITLAB_PAT}@gitlab.com>" > .git-credentials
# Tell git to use the credentials file
printf "[credential]\n\thelper = store\n" > .gitconfig
2) Create a Kubernetes Secret:
kubectl -n your-namespace create secret generic gitlab-https-creds \
--from-file=.git-credentials \
--from-file=.gitconfig
3) Mount them into the flow-run container via your Kubernetes work pool job template (or per-deployment job_variables in prefect.yaml). Example:
work_pool:
name: k8s-pool
job_variables:
image: <http://registry.gitlab.com/your-org/your-image:latest|registry.gitlab.com/your-org/your-image:latest>
pod:
volumes:
- name: git-creds
secret:
secretName: gitlab-https-creds
defaultMode: 0600
containers:
- name: prefect-job
volumeMounts:
- name: git-creds
mountPath: /root/.git-credentials
subPath: .git-credentials
readOnly: true
- name: git-creds
mountPath: /root/.gitconfig
subPath: .gitconfig
readOnly: true
Notes:
- Adjust paths if your image runs as a non-root user (e.g., /home/app).
- The flow-run image must have git installed.
B) Use a GitLab Deploy Token (scoped, recommended over user PATs)
- Create a Deploy Token with read_repository scope.
- Same mechanics as above, except your .git-credentials line will look like:
https://<deploy_token_username>:<deploy_token>@gitlab.com
- This avoids tying access to a human account and is easier to rotate.
Things not to do (unless you enjoy incident reports)
- Don’t bake the token into the deployment source URL (e.g., https://oauth2:TOKEN@gitlab.com/org/repo.git). Prefect will store deployment metadata; you don’t want your token in there.
- Don’t put the token in your image. Use Kubernetes Secrets and mount them at runtime.
Quick checklist
- Worker pool job template mounts the secret and your image has git + prefect + your deps.
- Your deployment’s source points to the HTTPS URL (no credentials in it), e.g.:
source="<https://gitlab.com/org/repo.git>"
entrypoint="flows/etl.py:etl_flow"
- Token rotation = update the Kubernetes Secret. No redeploy needed unless you hardcoded it somewhere (please don’t).
Docs for further comfort reading
- Store flow code in Git
- Kubernetes work pools and workers
If you share your namespace, pool name, and whether your container runs as root or a specific UID, I’ll give you a paste-ready job_variables block.Yu Shao Pang
08/11/2025, 4:35 PMMarvin
08/11/2025, 4:35 PMMarvin
08/11/2025, 4:35 PMMarvin
08/11/2025, 4:39 PMPREFECT_API_KEY
is only for Prefect Cloud. A self-hosted Prefect Server doesn’t use it.
For Server, point your client/agents at your API and (optionally) UI:
- Set API URL:
prefect config set PREFECT_API_URL=http://<host>:4200/api
- Optional UI URL (for links in logs, etc.):
prefect config set PREFECT_UI_URL=http://<host>:4200
- If you’re putting the API behind your own auth (reverse proxy), use:
prefect config set PREFECT_API_AUTH_STRING="Bearer <your-token>" # or "Basic <base64-creds>"
If you previously used Cloud, it’s safest to clear the key so nothing tries to “helpfully” authenticate to the wrong place:
prefect config unset PREFECT_API_KEY
If you need TLS knobs, there are settings for skipping verification or pointing at a cert file as well.Marvin
08/11/2025, 4:44 PMprefect cloud login -k <YOUR_API_KEY> -w <account_handle>/<workspace_handle>
- Change the current workspace later:
prefect cloud workspace set -w <account_handle>/<workspace_handle>
- Logout (clears API URL/key in the current profile):
prefect cloud logout
3) Set it via environment variable
- Unix/macOS:
export PREFECT_API_KEY="<YOUR_API_KEY>"
# Optional if you are not using `prefect cloud login`:
# export PREFECT_API_URL="<workspace API URL>"
- Windows (PowerShell):
$env:PREFECT_API_KEY="<YOUR_API_KEY>"
# Optional:
# $env:PREFECT_API_URL="<workspace API URL>"
Tip: You can persist it in your Prefect profile instead of env vars:
prefect config set PREFECT_API_KEY=<YOUR_API_KEY>
4) Programmatic use
- Most code should rely on your configured profile/env vars:
import asyncio
from prefect.client.orchestration import get_client
async def main():
async with get_client() as client:
# you're authenticated via PREFECT_API_KEY/profile here
...
asyncio.run(main())
- Or pass the key directly (and the API URL if needed):
import asyncio
from prefect.client.orchestration import PrefectClient
async def main():
async with PrefectClient(
api="<your workspace API URL>",
api_key="<YOUR_API_KEY>",
) as client:
...
asyncio.run(main())
Notes
- Local Prefect Server (self-hosted) doesn’t require an API key.
- Workers/runners in containers/CI typically set PREFECT_API_KEY
(and often PREFECT_API_URL
) via environment variables.
- You can view current settings and their sources:
prefect config view --show-sources
Docs
- Settings and profiles (includes PREFECT_API_KEY
): Prefect Settings and Profiles
- CLI reference: Prefect CLI
- Cloud UI (create API keys): Prefect Cloud
If you want, tell me your OS and whether you’re using a worker/runner, and I’ll give you the exact setup commands.Yu Shao Pang
08/11/2025, 4:44 PMMarvin
08/11/2025, 4:44 PMMarvin
08/11/2025, 4:48 PMYu Shao Pang
08/12/2025, 4:04 AMMarvin
08/12/2025, 4:04 AMMarvin
08/12/2025, 4:05 AM