<@ULVA73B9P> Is it possible to deploy two deployme...
# ask-marvin
v
@Marvin Is it possible to deploy two deployments that have the same name but different work pools ?
m
thought for 318.4 seconds
Short answer: Not within the same flow. In Prefect 3.x, a deployment’s name must be unique per flow; the work pool is not part of the uniqueness constraint. So creating two deployments for the same flow with the same name but different work pools is not allowed—the second “deploy” will update (upsert) the first. Details and source: - Uniqueness is on (flow_id, name), not work pool: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/server/database/orm_models.py#L904-L906 - Upsert key for deployments uses flow_id + name: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/server/database/orm_models.py#L1529-L1531 - API behavior: create will update an existing deployment with the same flow_id and name: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/server/api/deployments.py#L56-L73 - Model implementation of upsert: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/server/models/deployments.py#L87-L145 What you can do instead: - Give each pool-specific deployment a distinct name: - e.g.,
etl-ec2
and
etl-k8s
- Or, reuse the same deployment name across different flows (allowed because the uniqueness is per-flow) Example in code:
Copy code
from prefect import flow

@flow
def my_flow():
    ...

# Two deployments of the same flow to different work pools
my_flow.deploy(name="etl-ec2", work_pool_name="ec2-pool")
my_flow.deploy(name="etl-k8s", work_pool_name="k8s-pool")
If you instead try:
Copy code
my_flow.deploy(name="etl", work_pool_name="ec2-pool")
my_flow.deploy(name="etl", work_pool_name="k8s-pool")
the second call will update the original deployment to use
k8s-pool
. Does that fit your use case, or do you want help choosing a naming pattern for your pools/environments?
v
@Marvin Where does the flow_id come from ? My use case is that I have 2 distinct flows that I want to deploy to different work pools, and I find it practical to give deployments the name of the flow, so the deployments also have the same name. If I can get them to be considered as distinct deployments based on flow_id that would be great but I don't see where the flow_id is defined ?
m
thought for 344.8 seconds
Great question. In Prefect 3.x, the flow_id is a UUID created by the Prefect server when a Flow record is created. The key detail: Flow records are created/returned by name, and flow names are globally unique within your workspace. What that means in practice: - The flow_id is generated when the client calls create_flow_from_name(flow.name) during deployment. - Flow names are unique in the database, so two flows cannot share the same name; if you try, they’ll resolve to the same flow_id. - Deployment uniqueness is (flow_id, deployment_name). So you can reuse the same deployment name across different flows as long as the flows have different names. Where these rules come from (source links): - Flow name is unique (DB constraint): https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/server/database/orm_models.py#L152-L153 - Server create_flow logic returns existing flow if the name already exists (ON CONFLICT DO NOTHING): https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/server/models/flows.py#L20-L54 - Deployment uniqueness is (flow_id, name): https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/server/database/orm_models.py#L904-L906 and https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/server/database/orm_models.py#L1529-L1531 - API upsert behavior for deployments (same flow_id + name updates the existing deployment): https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/server/api/deployments.py#L56-L73 How to get what you want - If you have two distinct flows and you want to deploy them to different work pools but use the same deployment name, just give the flows different names. Then you can reuse the same deployment name across both. Example:
Copy code
from prefect import flow

@flow(name="orders")
def extract_orders():
    ...

@flow(name="customers")
def extract_customers():
    ...

# Same deployment name "prod" on two different flows -> OK
extract_orders.deploy(name="prod", work_pool_name="ec2-pool")
extract_customers.deploy(name="prod", work_pool_name="k8s-pool")
This works because the flows have different names, so they get different flow_ids. The deployments both named "prod" are distinct because uniqueness is (flow_id, "prod"). Tips - The flow name is taken from the @flow decorator. If you don’t set name=..., Prefect uses the function name. Set it explicitly if needed. - If two codebases define flows with the same name, Prefect will treat them as the same flow on the server (since flow names are unique). Rename one to avoid collisions. Useful CLI to inspect:
Copy code
prefect flow ls
prefect deployment ls --flow-name <flow-name>
If you share the two flow definitions (names) you’re using, I can suggest a clean naming pattern so you can reuse a single deployment name across them without collisions.
v
@Marvin Unrelated question, can I define the container name for docker runs ? Or at least a prefix to the container name ?
m
thought for 292.0 seconds
Yes—using the Docker work pool/worker in Prefect 3.x you can set the container name, but there is no built-in “prefix” option. How naming works: - If you set
job_variables.name
, Prefect uses that as the Docker container’s name (slugified to be Docker-safe). - If you don’t set it, Prefect falls back to the flow run’s name. - If a container with the same name already exists, Prefect automatically retries with -1, -2, etc. Where this comes from (source links): - Job config includes a
name
field used for container name: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/workers/base.py#L125-L131 - Base prep falls back to flow run name when
name
isn’t set: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/workers/base.py#L300 - Docker worker passes
name
to docker-py when creating the container and handles conflicts with -1, -2 suffixes: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/integrations/prefect-docker/prefect_docker/worker.py#L700-L717 and https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/integrations/prefect-docker/prefect_docker/worker.py#L889-L920 - Name is slugified for Docker compatibility: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/integrations/prefect-docker/prefect_docker/worker.py#L212-L237 - There’s no
name_prefix
feature anywhere in worker configs (confirmed by code search). How to set it - In code when deploying:
Copy code
my_flow.deploy(
  work_pool_name="my-docker-pool",
  job_variables={"name": "myapp-prod-{{ flow_run_id }}"}
)
- In prefect.yaml:
Copy code
deployments:
  - name: my-deployment
    work_pool:
      name: my-docker-pool
      job_variables:
        name: "etl-{{ flow_run_name }}"
- At run time via CLI:
Copy code
prefect deployment run "my-flow/my-deployment" --job-variable name=my-prefix-123
If you want a dynamic prefix, include it in the
name
template, e.g.
name: "teamA-{{ flow_run_name }}"
. Want help picking a stable template (e.g., including environment, project, or commit SHA)?
v
@Marvin I would like containers to be name "<flow_name>-<flow_run_name>"
m
thought for 398.7 seconds
Yes—you can get that result reliably by letting the Docker worker use the flow run’s name as the container name, then setting your flow run name to include the flow name. Why this works: - Docker worker uses
job_variables.name
if provided; otherwise it falls back to the flow run’s name. - Fallback behavior: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/prefect/workers/base.py#L300 - The Docker worker then slugifies the name and creates the container with it (and will append -1, -2 on conflicts): - Slugify + name usage: https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/integrations/prefect-docker/prefect_docker/worker.py#L212-L237 and https://github.com/PrefectHQ/prefect/blob/aa8034415f25a391ed7b8aa49d2acaebd1e559f3/src/integrations/prefect-docker/prefect_docker/worker.py#L889-L920 So, don’t set
job_variables.name
. Instead, set the flow’s
flow_run_name
to a callable that returns "<flow_name>-<whatever you want>". The container will inherit that exact value. Example:
Copy code
from prefect import flow, get_flow_run_context
import datetime

def container_like_flow_run_name():
    ctx = get_flow_run_context()
    # Safe unique suffix (first 8 chars of the run ID)
    suffix = str(ctx.flow_run.id)[:8]
    # Use the flow’s name and any parameter(s) you want
    # Here we combine flow name + an optional parameter + short id
    user = ctx.parameters.get("user", "anon")
    return f"{ctx.flow.name}-{user}-{suffix}"

@flow(flow_run_name=container_like_flow_run_name)
def my_flow(user: str = "anon"):
    ...

# Deploy without setting job_variables.name so container name == flow run name
# my_flow.deploy(work_pool_name="my-docker-pool")
Notes: - Job variable templating does not expose
flow
or
flow_run
objects, so setting
job_variables.name: "{{ flow.name }}-{{ flow_run.name }}"
won’t work. That’s why the callable
flow_run_name
is the right place to compose the name. - Docker will slugify the name (lowercase, replace spaces, etc.) and enforce uniqueness by appending
-1
,
-2
, etc. - Flow run name templating docs: https://docs.prefect.io/v3/how-to-guides/workflows/custom-metadata/ If you’d like, share your preferred run-name pattern and I’ll adapt the callable to match it (e.g., include env, date, short git SHA, etc.).