Short answer: - Different parameters: Yes — a single deployment can have multiple schedules, each with its own parameters. - Different work pools: No — a deployment can target only one work pool. To run on different pools, create separate deployments. Details (Prefect 3.x): -

Flow.deploy(...)

supports

schedules=[...]

where each

Schedule

can include its own

parameters

. -

work_pool_name

is a deployment-level setting, so it cannot vary by schedule. Example: one deployment, multiple schedules with different parameters (same work pool)

from prefect import Flow
from prefect.schedules import Schedule

flow = Flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/flow_file.py:my_flow",
)

flow.deploy(
    name="my-flow",
    work_pool_name="k8s-prod",  # single work pool for this deployment
    schedules=[
        Schedule(cron="0 2 * * *", timezone="UTC", parameters={"dataset": "A", "env": "prod"}),
        Schedule(cron="0 3 * * *", timezone="UTC", parameters={"dataset": "B", "env": "prod"}),
    ],
)

Example: different work pools → create separate deployments

flow = Flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/flow_file.py:my_flow",
)

# Deployment to k8s pool
flow.deploy(
    name="my-flow-k8s",
    work_pool_name="k8s-prod",
    schedules=[Schedule(cron="0 2 * * *", timezone="UTC", parameters={"dataset": "A"})],
)

# Deployment to ECS pool
flow.deploy(
    name="my-flow-ecs",
    work_pool_name="ecs-staging",
    schedules=[Schedule(cron="0 3 * * *", timezone="UTC", parameters={"dataset": "B"})],
)

Docs: - Schedules concepts: Schedules in Prefect 3 - How to add schedules to deployments: Create deployment schedules - Work pools overview: Work pools and workers blog If you share what pools and parameter sets you need, I can sketch the exact deployment code or CLI steps for your setup.

Great question. There are a few ways to run lots of deployments and “more than 10” logical lanes of work without racking up cost. Key idea: keep the number of work pools low and use work queues + job variables for segmentation - Use a small number of work pools per infrastructure type/environment (e.g., k8s-prod, k8s-staging, ecs-prod). A worker polls exactly one pool, so many pools = many always-on workers. - Create many work queues inside each pool—one per team/tenant/project/prioritization lane. Point deployments at queues via

work_queue_name

. - Use job variables on deployments to specialize infra (image, CPU/memory, env vars) instead of creating new pools. Benefits: - Fewer always-on workers → less infra overhead. - Per-queue concurrency limits give you tight cost control per tenant/team. - You still get clean routing with queues and deployment-level variables. Practical patterns 1) Create queues per team/tenant and cap concurrency

prefect work-queue create team-a --pool k8s-prod --limit 5
prefect work-queue create team-b --pool k8s-prod --limit 2

2) Start a single worker per pool, pull specific queues

prefect worker start --pool k8s-prod --work-queue team-a --work-queue team-b

3) Target queues from deployments

flow.deploy(
  name="etl-team-a",
  work_pool_name="k8s-prod",
  work_queue_name="team-a",
  job_variables={"image": "myrepo/etl:team-a", "cpu": "500m", "memory": "1Gi"},
)

4) Use pool- and queue-level limits to cap spend - Pool-wide cap:

prefect work-pool set-concurrency-limit k8s-prod 10

- Queue caps (shown above) let you protect shared pools from noisy neighbors. When you might really need many pools - You truly have different infra types or hard isolation requirements (e.g., separate k8s clusters, separate VPCs). If your plan has work pool limits or you must exceed them for isolation, reach out to us to discuss plan options so you’re not paying more than you need. If “expensive” means your compute bill - Run one minimal worker per pool (tiny CPU/mem) and push the heavy lifting into ephemeral job pods/containers. - Use spot/preemptible nodes where appropriate. - Right-size job variables per deployment so you don’t over-provision. Useful references - Work pools and workers overview: Work pools and workers - Customize job variables on deployments: Customize job variables - Work queue CLI (for limits/routing): run

prefect work-queue create --help

or see our CLI help in your terminal. Would you share: - What infrastructure types you’re using (k8s, ECS, Docker, process)? - Roughly how many tenants/teams and what isolation you require? - Whether the cost concern is Prefect Cloud plan limits or the underlying compute? With that, I can sketch an exact pool/queue layout and deployment config that minimizes steady-state cost.

You can reduce deployments by grouping clients per work queue and using schedules with parameters — or by using a “driver” pattern that spawns per-client runs every 5 minutes. Key constraints (Prefect 3.x): - A deployment can target only one work pool and one work queue. - A single deployment can have many schedules, and each Schedule can include its own

parameters

. Two effective patterns 1) One deployment per queue, many schedules (one per client) - You keep deployments to “# of queues,” not “# of clients x # of queues”. - Each schedule runs every 5 minutes and passes a different

client_id

. Example

from datetime import timedelta
from prefect import Flow
from prefect.schedules import Schedule

# group clients by the queue they should run on
clients_by_queue = {
    "queue-a": ["client_1", "client_2", "client_3"],
    "queue-b": ["client_4", "client_5"],
}

flow = Flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/flow_module.py:my_flow",
)

for queue_name, client_ids in clients_by_queue.items():
    schedules = [
        Schedule(
            interval=timedelta(minutes=5),
            parameters={"client_id": cid},
            slug=f"every-5m-{cid}",
        )
        for cid in client_ids
    ]

    flow.deploy(
        name=f"my-flow-{queue_name}",
        work_pool_name="shared-pool",
        work_queue_name=queue_name,
        schedules=schedules,
        # optional: cap this queue’s runs if you’re worried about load
        # concurrency_limit=10,
    )

Notes - This cuts deployment count to the number of queues. Each client is just a schedule entry. - If you have many clients, this can mean lots of schedules; it works, but see pattern 2 if it gets unwieldy. 2) One deployment per queue, single 5-min schedule, “driver” triggers per-client runs - The deployment runs every 5 minutes and triggers a separate flow run per client using the same deployment (and thus the same queue). - This keeps your schedule count tiny and scales well with many clients. Driver flow example

from datetime import timedelta
from prefect import flow, Flow
from prefect.schedules import Schedule
from prefect.deployments import run_deployment

@flow
def my_flow(client_id: str):
    # do work for this client_id
    ...

@flow
def driver(client_ids: list[str]):
    # fan out per-client runs on the same deployment/queue
    for cid in client_ids:
        run_deployment(
            name="my-flow-queue-a/my-flow-queue-a",  # <FLOW_NAME>/<DEPLOYMENT_NAME>
            parameters={"client_id": cid},
        )

# Deploy the worker flow (my_flow) once per queue
worker_flow = Flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/flow_module.py:my_flow",
)
worker_flow.deploy(
    name="my-flow-queue-a",
    work_pool_name="shared-pool",
    work_queue_name="queue-a",
)

# Deploy the driver once per queue with a single 5-min schedule
driver_flow = Flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/flow_module.py:driver",
)
driver_flow.deploy(
    name="driver-queue-a",
    work_pool_name="shared-pool",
    work_queue_name="queue-a",
    interval=timedelta(minutes=5),
    parameters={"client_ids": ["client_1", "client_2", "client_3"]},
)

Notes - The driver approach avoids thousands of schedules while still creating one run per client every 5 minutes. - Use pool/queue concurrency limits to control spend and avoid run storms: - Pool-wide:

prefect work-pool set-concurrency-limit <pool> <limit>

- Queue-wide:

prefect work-queue create <queue> --pool <pool> --limit <n>

Which should you choose? - Few-to-hundreds of clients per queue → Pattern 1 (per-client schedules) is simple and explicit. - Hundreds-to-thousands of clients per queue → Pattern 2 (driver) scales better and keeps the deployment’s schedule list small. Links

- Schedules in Prefect 3: Schedules in Prefect 3 - How to add schedules to deployments: Create deployment schedules - Customize job variables (if you need per-client images/env): Customize job variables Quick clarifiers to tailor the plan: - How many clients per queue are you targeting? - Do you need each client’s schedule visible in the UI, or is a single “driver” run that fans-out acceptable? - Any concurrency/SLA constraints per client or per queue?

Short answer: Not exactly. In Prefect 3.x, a deployment can only target one work queue. Schedules on a deployment can set different parameters, but they cannot change the queue. So you cannot have 1 deployment that schedules to 10 different queues. What you can do: - If one queue is acceptable: Use 1 deployment, 1 work pool, and add multiple schedules with different parameters (e.g., different client IDs) that all land on the same queue. - If you need different queues: Use 1 work pool and create 1 deployment per queue. Each deployment can still have many schedules with different parameters. Examples 1) One deployment, many schedules, same queue

from datetime import timedelta
from prefect import Flow
from prefect.schedules import Schedule

flow = Flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/flow.py:my_flow",
)

flow.deploy(
    name="my-flow",
    work_pool_name="shared-pool",
    work_queue_name="queue-a",  # one queue only
    schedules=[
        Schedule(interval=timedelta(minutes=5), parameters={"client_id": "A"}, slug="every-5m-A"),
        Schedule(interval=timedelta(minutes=5), parameters={"client_id": "B"}, slug="every-5m-B"),
        # add more clients here...
    ],
)

2) Different queues require separate deployments (still one work pool)

from datetime import timedelta
from prefect import Flow
from prefect.schedules import Schedule

flow = Flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/flow.py:my_flow",
)

for queue_name, client in [
    ("queue-a", "A"),
    ("queue-b", "B"),
    ("queue-c", "C"),
    # ...
]:
    flow.deploy(
        name=f"my-flow-{queue_name}",
        work_pool_name="shared-pool",
        work_queue_name=queue_name,
        schedules=[Schedule(interval=timedelta(minutes=5), parameters={"client_id": client})],
    )

Related docs - Schedules: Schedules in Prefect 3 - Creating schedules on deployments: Create deployment schedules If you share why each client needs its own queue (e.g., isolation or per-client concurrency), I can suggest alternatives like using per-queue deployments with concurrency limits, or a driver flow pattern to fan out runs efficiently.

Yes. With one deployment and many schedules (each with different parameters), you can monitor them cleanly in the UI with a couple of tweaks: What you’ll see in the UI - Deployment page → Schedules tab: Each schedule is listed separately. Give each schedule a clear slug (e.g., client-A-5m). You can pause/resume an individual client’s schedule here. - Runs page filtered to your deployment: Each run shows its parameters, state, and duration. Make each client easy to spot 1) Name each schedule with a client-specific slug

from datetime import timedelta
from prefect import Flow
from prefect.schedules import Schedule

flow = Flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/flow.py:my_flow",
)

flow.deploy(
    name="client-runner",
    work_pool_name="shared-pool",
    work_queue_name="queue-a",
    schedules=[
        Schedule(interval=timedelta(minutes=5), parameters={"client_id": "A"}, slug="client-A-5m"),
        Schedule(interval=timedelta(minutes=5), parameters={"client_id": "B"}, slug="client-B-5m"),
        # add more clients here...
    ],
)

2) Include the client_id in the flow run name so runs are searchable

from prefect import flow
from prefect.runtime import flow_run

def run_name():
    params = flow_run.parameters()
    return f"client-{params.get('client_id', 'unknown')}"

@flow(flow_run_name=run_name)
def my_flow(client_id: str):
    ...

- Now the UI run list will show names like client-A, client-B, etc., and you can quickly search by client in the run list. - Each run’s Parameters panel in the UI will also show the

client_id

. Tips - Use schedule slugs like client-A-5m so the Schedules tab is self-explanatory. - If you need to throttle per client, set a concurrency limit on the queue or the deployment to control spend. Docs - Schedules: Schedules in Prefect 3 - Creating deployment schedules: Create deployment schedules If you share your client list size, I can suggest whether per-client schedules or a “driver” flow pattern (one schedule that fans out per-client runs) will be easier to manage.

Short answer: You can’t target a specific worker directly. In Prefect 3.x, routing is done by work queues. To run client-specific schedules on particular workers, give each worker (or worker group) its own queue and point a deployment at that queue. A single deployment can’t send runs to different queues; that requires separate deployments. Recommended pattern - 1 work pool. - Create 1 queue per “target worker” (or worker group). - Start each worker to poll only its queue. - Create 1 deployment per queue. Inside each deployment, add multiple schedules with different client parameters. CLI setup

# queues for specific workers
prefect work-queue create worker-a-queue --pool shared-pool
prefect work-queue create worker-b-queue --pool shared-pool

# start workers pinned to their queues
prefect worker start --pool shared-pool --work-queue worker-a-queue --name worker-a
prefect worker start --pool shared-pool --work-queue worker-b-queue --name worker-b

Deployment(s)

from datetime import timedelta
from prefect import Flow
from prefect.schedules import Schedule

flow = Flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/flow.py:my_flow",
)

# Deployment to worker A’s queue (multiple clients, same queue)
flow.deploy(
    name="client-runner-worker-a",
    work_pool_name="shared-pool",
    work_queue_name="worker-a-queue",
    schedules=[
        Schedule(interval=timedelta(minutes=5), parameters={"client_id": "A1"}, slug="A1-5m"),
        Schedule(interval=timedelta(minutes=5), parameters={"client_id": "A2"}, slug="A2-5m"),
    ],
)

# Deployment to worker B’s queue (multiple clients, same queue)
flow.deploy(
    name="client-runner-worker-b",
    work_pool_name="shared-pool",
    work_queue_name="worker-b-queue",
    schedules=[
        Schedule(interval=timedelta(minutes=5), parameters={"client_id": "B1"}, slug="B1-5m"),
        Schedule(interval=timedelta(minutes=5), parameters={"client_id": "B2"}, slug="B2-5m"),
    ],
)

Why this is necessary - Queue selection is deployment-level. Schedules can change parameters but not the queue. - Workers only decide which runs to pick up by the queues they poll. There’s no “run on worker X” selector per run. If you have many clients per worker and want to keep schedules small, use a “driver” deployment per queue: one 5-minute schedule that calls

run_deployment(...)

to fan out per-client runs for that queue. This keeps the number of schedules low while still routing to the right worker via the deployment’s queue. Helpful docs - Work pools and workers: Work pools and workers - Create deployment schedules: Create deployment schedules If you can share how many workers/queues and clients you have, I can sketch the exact deployment/queue layout that minimizes the number of deployments while keeping routing correct.

You’re right — when you create a flow run programmatically (or via the UI) from a deployment, you can override both parameters and the work queue for that single run. That’s exactly what

run_deployment(...)

supports. Key distinction: - Ad-hoc/programmatic runs: You can override

work_queue_name

per run. - Deployment schedules: You cannot set a different queue per schedule; schedules always use the deployment’s queue. Ad-hoc example (override queue + parameters)

from prefect.deployments import run_deployment

run_deployment(
    name="my-flow/my-deployment",
    parameters={"client_id": "A"},
    work_queue_name="worker-a-queue",  # per-run queue override
)

API docs: run_deployment API If you need “scheduled but different queues” from one deployment: - Use a driver flow that runs every 5 minutes and calls

run_deployment(...)

multiple times, setting both the client-specific parameters and the target queue for each run. Driver pattern

from datetime import timedelta
from prefect import flow, Flow
from prefect.schedules import Schedule
from prefect.deployments import run_deployment

@flow
def driver():
    plan = [
        {"client_id": "A", "queue": "worker-a-queue"},
        {"client_id": "B", "queue": "worker-b-queue"},
        # ...
    ]
    for item in plan:
        run_deployment(
            name="my-flow/my-deployment",
            parameters={"client_id": item["client_id"]},
            work_queue_name=item["queue"],
        )

# Deploy the driver on any queue; it fans out runs to specific queues
driver_flow = Flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/driver.py:driver",
)
driver_flow.deploy(
    name="driver-5m",
    work_pool_name="shared-pool",
    work_queue_name="control-queue",
    schedules=[Schedule(interval=timedelta(minutes=5))],
)

This gives you: - One deployment for the actual work. - One scheduled driver deployment that fans out per-client runs to specific queues using per-run queue overrides. If you prefer not to use a driver, the only alternative is separate deployments per queue (still one work pool), each with schedules for their clients.

Great question — here’s a balanced comparison of the two patterns in Prefect 3.x. Single deployment + many schedules (different parameters per client) Pros - Minimal management: one artifact to build and update. - Easy onboarding: add a schedule entry per new client. - Uniform runtime: all clients share the same image and infrastructure. Cons - One queue only: schedules can’t target different queues; all runs land on the deployment’s queue. - Shared infra settings: one set of job variables (image, CPU/mem, env) for everyone; you can’t size clients differently. - Lockstep versioning: changing the deployment updates all clients at once; you can’t pin some clients to older versions. - Limited per-client controls: - Concurrency limit is deployment-level, not per schedule/client. - Harder to do per-client notifications/RBAC without extra tagging logic. - UI scale/clarity: many client schedules in one deployment can get crowded; pausing all runs for one client = pausing just that client’s schedule, which is okay but less obvious than pausing a dedicated deployment. Separate deployment per client Pros - Routing flexibility: each client can target its own queue/worker(s). - Isolation: per-client job variables (image/resources), secrets, tags, and concurrency caps. - Independent versioning: roll out or pin versions by client. - Cleaner operations: per-client pause/resume, RBAC, notifications, dashboards. Cons - More to manage: more deployments to create, update, and monitor. - CI/CD overhead: you’ll likely script/automate deployment creation. Good middle ground patterns - One deployment per queue/worker group: put all clients that share the same infrastructure/priority into a single queue; give that queue one deployment with multiple schedules. This yields: - Per-queue concurrency and worker isolation - Fewer deployments than “one per client” - Driver pattern for targeted queues with one “work” deployment: - A scheduled “driver” flow runs every 5 min and calls

run_deployment(...)

per client, overriding both parameters and

work_queue_name

per run. This keeps deployment and schedule counts small but still routes to different workers. - API: run_deployment When to choose which - Choose “single deployment + many schedules” if: - All clients can share the same queue, image, and resources - You’re okay with lockstep upgrades - You have tens (not thousands) of clients and want simplicity - Choose “separate deployment per client (or per queue)” if: - You need client-specific queues/workers/resources - You want per-client concurrency limits, alerts, and versioning - You need strong isolation and clearer UI/RBAC boundaries Docs - Schedules: Schedules in Prefect 3 - Create schedules on deployments: Create deployment schedules - Customize job variables: Customize job variables - Work pools/queues overview: Work pools and workers If you share: - How many clients - Whether they need different workers/resources - Any per-client concurrency/SLA requirements I can suggest an exact layout (single, per-queue, or driver) that minimizes deployments while keeping the routing and controls you need.

Yes—it will work. Separate deployment per client is fully supported on self-hosted Prefect Server (on‑prem). Orchestration concepts are the same: deployments, schedules, work pools, work queues, and workers all behave like in Cloud. What’s different vs Prefect Cloud - You run the control plane: - You manage the API/UI, database, backups, upgrades, and scaling of the scheduler/services. - Capacity planning matters if you have lots of deployments/schedules (e.g., many 5‑min schedules). - Auth and org features: - Cloud provides SSO, RBAC, orgs/workspaces, API key management, and audit logs. On Server, these are limited or DIY. - Automations and incidents: - Cloud has managed Automations (state/metric triggers, notifications) and Incidents. On Server, you’ll rely more on in‑flow logic, webhooks, or external tooling for alerts/ops. - Governance and convenience: - Cloud adds richer multi‑workspace separation and policy controls; Server gives you full control but you build more around it. - Secrets/security: - Both support Blocks/Variables; in Server you configure storage/encryption yourself and ensure network/security posture. Good practices for “one deployment per client” on Server - Automate deployment creation so you don’t manage them by hand:

from prefect import Flow
clients = ["client_a", "client_b", "client_c"]

flow = Flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/flow.py:my_flow",
)

for cid in clients:
    flow.deploy(
        name=f"my-flow-{cid}",
        work_pool_name="shared-pool",
        work_queue_name="queue-shared-or-per-client",
        # Optional: per-client schedule and resources
        # schedules=[...],
        # job_variables={"image": "...", "cpu": "...", "memory": "..."},
        parameters={"client_id": cid},
    )

- Use queues and limits to control spend and isolation: - Put clients on shared or per-client queues as needed and set queue/pool concurrency limits. - Make runs easy to find in the UI: - Include the client id in

flow_run_name

and/or tags so you can filter quickly. - Keep the DB healthy: - With many frequent schedules, monitor DB size, tune resources, and periodically clean up old runs/logs based on your retention policy (you can script cleanup with the CLI/API). Where to read more - Schedules: Schedules in Prefect 3 - Create deployment schedules: Create deployment schedules - Customize job variables: Customize job variables - Work pools/queues overview: Work pools and workers If you share roughly how many clients and how often they run, I can suggest a scaling plan for your Server setup (DB sizing, worker layout, queues, and concurrency limits).

Great questions — here’s how I’d design this in Prefect 3.x, plus tradeoffs and what changes if you’re building your own Next.js UI. 1) Building a Next.js “admin” UI to manage Prefect - Yes, it’s feasible. Use server-side routes (API routes in Next.js) to call the Prefect API with a service account/API key. Don’t put your Prefect API key in the browser. - What your UI can safely do via the API: - List flows/deployments/work pools/queues and flow runs - Create ad-hoc runs with parameters, override work_queue per-run - Pause/resume a deployment’s schedules (and individual schedules if you model them) - Set concurrency limits on work pools/queues - Link out to the Prefect UI for deep inspection of runs/logs - What’s better handled by CI/CD or a backend worker service: - Creating/updating deployments end-to-end. The Prefect API does not build images or package your code. Typically you: - Trigger a CI job from your Next.js UI to run

prefect deploy

(or call a Python service that runs

Flow.from_source(...).deploy(...)

) - Store Git refs/images and job variables per client - Monitoring from your UI: - Poll Prefect’s runs endpoints and show status, parameters, durations - Optionally listen for events/logs if you want “live” updates (proxy through your server to protect tokens) - Useful API in code for ad-hoc runs: -

run_deployment(...)

lets you override parameters and work_queue per run: run_deployment API 2) One deployment with many schedules vs per-client deployments - Use one deployment with many schedules if: - All clients can share the same queue/worker(s), image, and resources - You’re okay with lockstep upgrades (all clients move together) - You want minimal deployment count - Use one deployment per client (or per queue) if: - You need specific routing (client → specific worker/queue) - You want per-client job variables (image/CPU/memory/env), concurrency, and version pinning - You want clearer per-client pause/resume, RBAC, notifications, and dashboards - Middle ground that scales well: - Per-queue deployments: group similar clients into a queue with one deployment and many schedules - Driver pattern: one scheduled “driver” flow that runs every 5 minutes and calls

run_deployment(...)

per client, overriding both parameters and work_queue per run (keeps schedule lists small) 3) “Deploy agent from UI to infra and manage from UI” - In Prefect 3.x, agents were replaced by workers. - Starting a worker is an infrastructure action (a process/container on your infra). Prefect’s UI cannot start a process on your cluster directly. - Two good approaches: - Infra-managed workers (recommended, predictable): - Use Helm/Terraform/ECS/Kustomize to deploy worker containers in your environments. - From your Next.js UI, trigger your infra pipeline to “install worker X” on a given cluster (e.g., a Kubernetes Deployment or ECS Service running

prefect worker start --pool <pool> --work-queue <queue>

). - Manage everything else (concurrency limits, queues, schedules) via Prefect API and UI. - Work Pool provisioners (where available): - Prefect has work pool “provisioners” for some infra types to help bootstrap worker infra. You’d still need cloud permissions and must manage lifecycle, but this can simplify setup. In practice most teams still wrap this in Terraform/Helm. - Managing from the UI: - Use Prefect’s UI for visibility, pausing/resuming pools/queues/schedules, inspecting runs/logs. - Your Next.js UI can provide higher-level controls (e.g., “Install worker for Client A”, “Pause Client A”), which under the hood: - Calls your infra pipeline for worker lifecycle - Calls Prefect API for pools/queues/schedules and run orchestration CLI and code you’ll likely wire up behind your UI - Create queues and set limits ```

prefect work-queue create client-a --pool shared-pool --limit 5 prefect work-queue create client-b --pool shared-pool --limit 2

- Start workers pinned to queues (your infra runs this as a container/process)

prefect worker start --pool shared-pool --work-queue client-a --name worker-client-a

- Create runs from a deployment with overrides

from prefect.deployments import run_deployment run_deployment( name="my-flow/my-deployment", parameters={"client_id": "client-a"}, work_queue_name="client-a", # per-run routing ) ``` Recommendation for your setup - If clients need specific workers/queues or resources, go with per-client deployments or per-queue deployments: - One work pool per environment/infra type (e.g., k8s-prod) - One queue per client (or per client group) - One deployment per queue, with schedules per client OR a driver to fan out runs - Build a small backend service (invoked by your Next.js UI) that: - Triggers CI to run

prefect deploy

for new/updated deployments - Calls Prefect API to manage queues, limits, schedules, and ad-hoc runs - Proxies read operations (list runs, deployments) for your UI - Use Prefect UI alongside your Next.js app for deep inspection of runs/logs, while your app focuses on high-level operations and self-service. Helpful links - Schedules overview: Schedules in Prefect 3 - Create deployment schedules: Create deployment schedules - Work pools and workers: Work pools and workers - Programmatically creating runs: run_deployment API If you share: - Infra target (K8s/ECS/Docker/VMs) - # of clients and how often they run - Whether clients need different images/resources/queues I can sketch a concrete plan (queues, deployments, limits) and the minimal API endpoints your Next.js backend should implement.

Yes, within a single deployment you can use a

client_id

parameter to customize what the flow does for specific clients. This works great for logic-level customization (which tasks run, modes like full vs incremental, feature toggles, per-client concurrency). It does not change infrastructure (image/CPU/memory) or routing to different workers/queues from a schedule. What you can customize safely in one deployment - Branching/feature toggles by client_id - Per-client concurrency using tags/concurrency limits - Run naming and tagging for easy monitoring and filtering Example: client-specific logic, tags, and per-client concurrency

from prefect import flow, tags
from prefect.runtime import flow_run
from prefect.variables import Variable
from prefect.concurrency.sync import concurrency

def run_name():
    params = flow_run.parameters()
    return f"client-{params.get('client_id','unknown')}"

@flow(flow_run_name=run_name)
def my_flow(client_id: str):
    # Read a list of “premium” clients from a Prefect Variable (JSON list)
    premium_clients = set(Variable.get("premium_clients", default=[]))
    is_premium = client_id in premium_clients

    # Tag runs so you can filter/monitor per client in the UI
    with tags(f"client:{client_id}", "premium" if is_premium else "standard"):
        # Optional: enforce per-client concurrency with a tag-based limit
        # First, set a limit: `prefect concurrency-limit create client:clientA 1`
        with concurrency(f"client:{client_id}", occupy=1):
            if is_premium:
                # premium path: extra tasks, full refresh, etc.
                ...
            else:
                # standard path
                ...

Scheduling multiple clients from one deployment (same queue)

from datetime import timedelta
from prefect import Flow
from prefect.schedules import Schedule

flow = Flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="path/to/flow.py:my_flow",
)

flow.deploy(
    name="client-runner",
    work_pool_name="shared-pool",
    work_queue_name="queue-a",
    schedules=[
        Schedule(interval=timedelta(minutes=5), parameters={"client_id": "A"}, slug="A-5m"),
        Schedule(interval=timedelta(minutes=5), parameters={"client_id": "B"}, slug="B-5m"),
    ],
)

What a single deployment cannot change per client (via schedules) - Queue/worker routing: schedules always use the deployment’s queue. If you need client-specific workers, use: - separate deployments per queue/client, or - a “driver” flow that calls

run_deployment(...)

and overrides

work_queue_name

per run: run_deployment API - Infrastructure (image/CPU/memory/job variables): deployment-level by default. You can override per-run only when creating runs programmatically via

run_deployment(..., job_variables=...)

, not from schedules: Customize job variables Docs - Schedules: Schedules in Prefect 3 - Create schedules: Create deployment schedules Quick check: Do you need to customize infrastructure/routing per client, or just change execution logic? If it’s logic-only, a single deployment with client_id-based branching works well. If you need queue/worker or resources to differ per client, use per-client (or per-queue) deployments or the driver pattern.