Vincent Chéry
12/05/2025, 12:37 PMMarvin
12/05/2025, 12:37 PMMarvin
12/05/2025, 12:42 PMetl-ec2 and etl-k8s
- Or, reuse the same deployment name across different flows (allowed because the uniqueness is per-flow)
Example in code:
from prefect import flow
@flow
def my_flow():
...
# Two deployments of the same flow to different work pools
my_flow.deploy(name="etl-ec2", work_pool_name="ec2-pool")
my_flow.deploy(name="etl-k8s", work_pool_name="k8s-pool")
If you instead try:
my_flow.deploy(name="etl", work_pool_name="ec2-pool")
my_flow.deploy(name="etl", work_pool_name="k8s-pool")
the second call will update the original deployment to use k8s-pool.
Does that fit your use case, or do you want help choosing a naming pattern for your pools/environments?Vincent Chéry
12/05/2025, 1:01 PMMarvin
12/05/2025, 1:01 PMMarvin
12/05/2025, 1:07 PMfrom prefect import flow
@flow(name="orders")
def extract_orders():
...
@flow(name="customers")
def extract_customers():
...
# Same deployment name "prod" on two different flows -> OK
extract_orders.deploy(name="prod", work_pool_name="ec2-pool")
extract_customers.deploy(name="prod", work_pool_name="k8s-pool")
This works because the flows have different names, so they get different flow_ids. The deployments both named "prod" are distinct because uniqueness is (flow_id, "prod").
Tips
- The flow name is taken from the @flow decorator. If you don’t set name=..., Prefect uses the function name. Set it explicitly if needed.
- If two codebases define flows with the same name, Prefect will treat them as the same flow on the server (since flow names are unique). Rename one to avoid collisions.
Useful CLI to inspect:
prefect flow ls
prefect deployment ls --flow-name <flow-name>
If you share the two flow definitions (names) you’re using, I can suggest a clean naming pattern so you can reuse a single deployment name across them without collisions.Vincent Chéry
12/05/2025, 1:15 PMMarvin
12/05/2025, 1:15 PMMarvin
12/05/2025, 1:20 PMjob_variables.name, Prefect uses that as the Docker container’s name (slugified to be Docker-safe).
- If you don’t set it, Prefect falls back to the flow run’s name.
- If a container with the same name already exists, Prefect automatically retries with -1, -2, etc.
Where this comes from (source links):
- Job config includes a name field used for container name: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/workers/base.py#L125-L131
- Base prep falls back to flow run name when name isn’t set: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/workers/base.py#L300
- Docker worker passes name to docker-py when creating the container and handles conflicts with -1, -2 suffixes: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/integrations/prefect-docker/prefect_docker/worker.py#L700-L717 and https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/integrations/prefect-docker/prefect_docker/worker.py#L889-L920
- Name is slugified for Docker compatibility: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/integrations/prefect-docker/prefect_docker/worker.py#L212-L237
- There’s no name_prefix feature anywhere in worker configs (confirmed by code search).
How to set it
- In code when deploying:
my_flow.deploy(
work_pool_name="my-docker-pool",
job_variables={"name": "myapp-prod-{{ flow_run_id }}"}
)
- In prefect.yaml:
deployments:
- name: my-deployment
work_pool:
name: my-docker-pool
job_variables:
name: "etl-{{ flow_run_name }}"
- At run time via CLI:
prefect deployment run "my-flow/my-deployment" --job-variable name=my-prefix-123
If you want a dynamic prefix, include it in the name template, e.g. name: "teamA-{{ flow_run_name }}". Want help picking a stable template (e.g., including environment, project, or commit SHA)?Vincent Chéry
12/05/2025, 1:34 PMMarvin
12/05/2025, 1:34 PMMarvin
12/05/2025, 1:40 PMjob_variables.name if provided; otherwise it falls back to the flow run’s name.
- Fallback behavior: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/workers/base.py#L300
- The Docker worker then slugifies the name and creates the container with it (and will append -1, -2 on conflicts):
- Slugify + name usage: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/integrations/prefect-docker/prefect_docker/worker.py#L212-L237 and https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/integrations/prefect-docker/prefect_docker/worker.py#L889-L920
So, don’t set job_variables.name. Instead, set the flow’s flow_run_name to a callable that returns "<flow_name>-<whatever you want>". The container will inherit that exact value.
Example:
from prefect import flow, get_flow_run_context
import datetime
def container_like_flow_run_name():
ctx = get_flow_run_context()
# Safe unique suffix (first 8 chars of the run ID)
suffix = str(ctx.flow_run.id)[:8]
# Use the flow’s name and any parameter(s) you want
# Here we combine flow name + an optional parameter + short id
user = ctx.parameters.get("user", "anon")
return f"{ctx.flow.name}-{user}-{suffix}"
@flow(flow_run_name=container_like_flow_run_name)
def my_flow(user: str = "anon"):
...
# Deploy without setting job_variables.name so container name == flow run name
# my_flow.deploy(work_pool_name="my-docker-pool")
Notes:
- Job variable templating does not expose flow or flow_run objects, so setting job_variables.name: "{{ flow.name }}-{{ flow_run.name }}" won’t work. That’s why the callable flow_run_name is the right place to compose the name.
- Docker will slugify the name (lowercase, replace spaces, etc.) and enforce uniqueness by appending -1, -2, etc.
- Flow run name templating docs: https://docs.prefect.io/v3/how-to-guides/workflows/custom-metadata/
If you’d like, share your preferred run-name pattern and I’ll adapt the callable to match it (e.g., include env, date, short git SHA, etc.).