https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> what’s the best practice to add/appl...
# ask-marvin
a
@Marvin what’s the best practice to add/apply Prefect deployments while serving prefect (server) and prefect-worker from a single docker-compose file? context: repo tracks prefect.yaml; docker-compose runs server + worker; data scientists will trigger runs from the UI. Goal: deployments exist and are runnable, no secrets in repo, idempotent and reproducible behavior
m
✅ thought for 624.9 seconds
Great question — here’s a pattern that hits all your goals (Prefect 3.x), works cleanly with a single docker-compose, and keeps secrets out of the repo: High-level best practice - Track deployments in 
prefect.yaml
and register them with 
prefect deploy
during container startup (idempotent). - Run a short “bootstrap” init container that: - Ensures your work pool exists - Seeds Variables/Secrets from environment (not git) - Runs 
prefect deploy
to upsert deployments from 
prefect.yaml
- Run a single worker that polls that pool. Data scientists trigger runs from the UI. - Pull code at runtime via the 
git_clone
pull step (or use pre-built images if you prefer a Docker worker). Important notes for Prefect 3.x - Do NOT use 
prefect deployment build
or 
Deployment.build_from_flow()
— both are removed. Use 
prefect deploy
or 
flow.from_source(...).deploy(...)
. - Prefer a process worker for simplicity in docker-compose; switch to a Docker/Kubernetes worker later if you want container-per-run isolation. Minimal, reproducible setup 1) prefect.yaml (tracked in your repo) - Declaratively define deployments and a pull step to fetch code at run time. No secrets here.
Copy code
name: my-project
prefect-version: 3.0.0

# Pull code for each run
pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://github.com/my-org/my-repo.git>
      branch: "{{ prefect.variables.deployment_branch }}"

deployments:
  - name: hello-deployment
    entrypoint: flows/hello.py:hello_flow
    work_pool:
      name: process-pool
    parameters:
      name: "world"
    tags: ["production"]
    schedules:
      - cron: "0 9 * * *"
        timezone: UTC
- For private repos, add credentials to the 
git_clone
step per docs and store tokens as a Secret/Variable (never commit them). See Store flow code. 2) docker-compose.yml - One service for server, a short-lived bootstrap init, and the worker.
Copy code
version: '3.8'

services:
  prefect-server:
    image: prefecthq/prefect:3-latest
    command: prefect server start
    environment:
      PREFECT_UI_ENABLED: "true"
    ports:
      - "4200:4200"
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request as u; u.urlopen('<http://localhost:4200/api/health>', timeout=1)"]
      interval: 10s
      timeout: 5s
      retries: 12

  bootstrap:
    image: prefecthq/prefect:3-latest
    depends_on:
      prefect-server:
        condition: service_healthy
    environment:
      PREFECT_API_URL: <http://prefect-server:4200/api>
      # Feed secrets/values in via your .env (not committed)
      DEPLOYMENT_BRANCH: ${DEPLOYMENT_BRANCH:-main}
      API_KEY: ${API_KEY}
      DATABASE_URL: ${DATABASE_URL}
    working_dir: /app
    volumes:
      - ./prefect.yaml:/app/prefect.yaml
      - ./bootstrap.py:/app/bootstrap.py
    command: >
      sh -c "
        python bootstrap.py &&
        prefect deploy
      "
    restart: on-failure

  prefect-worker:
    image: prefecthq/prefect:3-latest
    depends_on:
      bootstrap:
        condition: service_completed_successfully
    environment:
      PREFECT_API_URL: <http://prefect-server:4200/api>
      # These will be used at runtime by flows if needed
      API_KEY: ${API_KEY}
      DATABASE_URL: ${DATABASE_URL}
    command: prefect worker start --pool process-pool
    restart: unless-stopped
- This uses defaults and avoids maintaining Postgres/Redis unless you need them. If you want Postgres/Redis for higher throughput, add those services as in Prefect’s self-hosted guide. - If you prefer builds/pushes and a Docker worker, add 
prefect-docker
to your project, mount the Docker socket into the worker, and use build/push steps in 
prefect.yaml
. 3) bootstrap.py (idempotent) - Ensures your work pool exists, sets Variables/Secrets from env (safe to re-run), then 
prefect deploy
upserts deployments. ``` import asyncio import os
from prefect.client.orchestration import get_client from prefect.variables import Variable from prefect.blocks.system import Secret POOL_NAME = "process-pool" async def main(): async with get_client() as client: # Ensure a process-type pool exists (idempotent via overwrite in CLI or upsert here) try: await client.read_work_pool(POOL_NAME) print(f"✓ Work pool '{POOL_NAME}' exists") except Exception: # Fallback to CLI creation for clarity; this is safe to run multiple times with --overwrite # If you prefer pure Python, you can use client APIs to create a process pool. import subprocess subprocess.run( ["prefect", "work-pool", "create", POOL_NAME, "--type", "process", "--overwrite"], check=True, ) print(f"✓ Work pool '{POOL_NAME}' created") # Variables (overwrite=True makes this idempotent) Variable.set("deployment_branch", os.getenv("DEPLOYMENT_BRANCH", "main"), overwrite=True) # Secrets from env -> Secret blocks (idempotent with overwrite) env_secrets = { "api_key": os.getenv("API_KEY"), "database_url": os.getenv("DATABASE_URL"), } for name, value in env_secrets.items(): if value: Secret(value=value).save(name, overwrite=True) print(f"✓ Secret '{name}' saved") if name == "__main__": asyncio.run(main()) ``` Why this meets your goals - Idempotent and reproducible: bootstrap can run on every up, 
prefect deploy
upserts deployments, Variables/Secrets set with overwrite. - No secrets in repo: everything sensitive comes from environment (.env into docker-compose), is set as Variables/Secret blocks, and referenced by code or 
prefect.yaml
templates. - UI-triggered runs: deployments are registered before the worker starts; they’ll show up in the UI and run on the worker. - Single compose file: server, bootstrap, worker all defined together with clear dependencies. Common variations - Process worker (shown) vs Docker worker: process is simpler; Docker worker provides container-per-run isolation. If you choose Docker worker, add 
prefect-docker
to your image or requirements and use build/push/image fields in 
prefect.yaml
. See Define deployments with prefect.yaml. - Code storage: use 
git_clone
pull step (shown) or 
flow.from_source(...).deploy(...)
if you prefer Python-based registration. See Deploy via Python and Store flow code. Verified docs to dig deeper - Create deployments - Define deployments with prefect.yaml - Store flow code (git clone step) - Variables - Store secrets - Self-host with docker-compose Questions to tailor this for you - Are you planning to use a process worker (simplest) or a Docker worker (container-per-run)? - Is your repo private? If so, do you want to authenticate via a Secret token or SSH key? If you share those, I can tweak the snippets (e.g., add credentials to the 
git_clone
step or switch to a Docker worker setup).
a
I am going to use process worker, and repo is private @Marvin
m
✅ thought for 200.2 seconds
Great — with a process worker and a private repo, you’ve got two solid, reproducible options. I’d default to SSH deploy keys for simplicity and stability in docker-compose; PATs with provider credential blocks are also first-class. Option A (recommended): SSH deploy key (no extra packages/blocks) - Create a read-only deploy key on your Git host and add it to the repo. - Put the private key + known_hosts on the host and mount them read-only into the worker. - Use an SSH repo URL in 
prefect.yaml
and the 
git_clone
pull step — no secrets in the repo, no blocks needed. prefect.yaml (SSH)
Copy code
name: my-project
prefect-version: 3.0.0

pull:
  - prefect.deployments.steps.git_clone:
      repository: git@github.com:org/private-repo.git
      branch: "{{ prefect.variables.deployment_branch }}"

deployments:
  - name: hello-deployment
    entrypoint: flows/hello.py:hello_flow
    work_pool:
      name: process-pool
docker-compose (worker w/ SSH mount)
Copy code
services:
  prefect-worker:
    image: prefecthq/prefect:3-latest
    environment:
      PREFECT_API_URL: <http://prefect-server:4200/api>
    command: prefect worker start --pool process-pool
    volumes:
      - ~/.ssh:/home/prefect/.ssh:ro
    user: "prefect"
    restart: unless-stopped
Notes - Ensure 
~/.ssh
contains the deploy key (chmod 600) and a 
known_hosts
entry for your host to avoid prompts:
Copy code
ssh-keyscan -t rsa <http://github.com|github.com> >> ~/.ssh/known_hosts
- If your base image doesn’t include git/ssh, use a small derived image to 
apt-get install -y git openssh-client
. Option B: Provider token via a credentials block (GitHub shown) - Keep token out of the repo: pass via 
.env
-> docker-compose -> bootstrap. - Register the integration’s block types, create a credentials block once in bootstrap, and reference it from 
prefect.yaml
. prefect.yaml (GitHub token via block)
Copy code
name: my-project
prefect-version: 3.0.0

pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://github.com/org/private-repo.git>
      branch: "{{ prefect.variables.deployment_branch }}"
      credentials: "{{ prefect.blocks.github-credentials.my-github-creds }}"

deployments:
  - name: hello-deployment
    entrypoint: flows/hello.py:hello_flow
    work_pool:
      name: process-pool
bootstrap.py (idempotent: pool + variable + block)
Copy code
import asyncio, os, subprocess
from prefect.client.orchestration import get_client
from prefect.variables import Variable

POOL = "process-pool"

async def main():
    async with get_client() as client:
        try:
            await client.read_work_pool(POOL)
            print(f"✓ Work pool '{POOL}' exists")
        except Exception:
            subprocess.run(
                ["prefect","work-pool","create",POOL,"--type","process","--overwrite"],
                check=True,
            )
            print(f"✓ Work pool '{POOL}' created")

    Variable.set("deployment_branch", os.getenv("DEPLOYMENT_BRANCH","main"), overwrite=True)

    # Register block types (safe to re-run)
    subprocess.run(["pip","install","prefect[github]"], check=True)
    subprocess.run(["prefect","block","register","-m","prefect_github"], check=True)

    from prefect_github import GitHubCredentials
    token = os.getenv("GITHUB_TOKEN")
    if token:
        GitHubCredentials(token=token).save("my-github-creds", overwrite=True)
        print("✓ GitHub credentials block saved")

if __name__ == "__main__":
    asyncio.run(main())
docker-compose (bootstrap + worker) ``` services: bootstrap: image: prefecthq/prefect:3-latest depends_on: prefect-server: condition: service_healthy environment: PREFECT_API_URL: http://prefect-server:4200/api DEPLOYMENT_BRANCH: ${DEPLOYMENT_BRANCH:-main} GITHUB_TOKEN: ${GITHUB_TOKEN} working_dir: /app volumes: - ./prefect.yaml:/app/prefect.yaml - ./bootstrap.py:/app/bootstrap.py command: > sh -c " python bootstrap.py && prefect deploy
" restart: on-failure prefect-worker: image: prefecthq/prefect:3-latest depends_on: bootstrap: condition: service_completed_successfully environment: PREFECT_API_URL: http://prefect-server:4200/api command: prefect worker start --pool process-pool restart: unless-stopped ``` Why these meet your goals - Deployments “exist and are runnable”: 
bootstrap
upserts the work pool/blocks/variables, runs 
prefect deploy
. - No secrets in repo: secrets come from env or mounted SSH keys; nothing committed. - Idempotent and reproducible: re-running 
bootstrap
is safe (
overwrite=True
, 
--overwrite
, and 
prefect deploy
is upsert-y). - DS can trigger from UI once compose is up. Pick your provider - If it’s GitHub and you want tokens: use Option B as written. - If it’s GitLab/Bitbucket, swap to 
prefect_gitlab
/ 
prefect_bitbucket
credentials blocks and register them similarly. - If you prefer the simplest runtime with the fewest moving parts, Option A (SSH deploy key) is great for a single process worker host. Docs to reference - Define deployments with prefect.yaml - Store flow code (git_clone for private repos) - Variables - Store secrets - Self-host with docker-compose If you tell me your Git provider (GitHub/GitLab/Bitbucket), I can drop in the exact block registration and 
prefect.yaml
snippet for it.
a
I am facing this issue @Marvin, can you help with this 021525.270 | ERROR | Task run 'trigger-dbt-transformation-6bd' - Task run failed with exception: ConnectError('[Errno -2] Name or service not known') Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions yield File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 394, in handle_async_request resp = await self._pool.handle_async_request(req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request raise exc from None File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request response = await connection.handle_async_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request raise exc File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 78, in handle_async_request stream = await self._connect(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 124, in _connect stream = await self._network_backend.connect_tcp(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp return await self._backend.connect_tcp( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 113, in connect_tcp with map_exceptions(exc_map): File "/usr/local/lib/python3.11/contextlib.py", line 158, in exit self.gen.throw(typ, value, traceback) File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions raise to_exc(exc) from exc httpcore.ConnectError: [Errno -2] Name or service not known The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/prefect/task_engine.py", line 1610, in run_context yield self File "/usr/local/lib/python3.11/site-packages/prefect/task_engine.py", line 1689, in run_task_async await engine.call_task_fn(txn) File "/usr/local/lib/python3.11/site-packages/prefect/task_engine.py", line 1627, in call_task_fn result = await call_with_parameters(self.task.fn, parameters) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/src/iw_data_pipeline/tasks/dbt.py", line 22, in trigger_dbt_job return await trigger_dbt_job_service( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/src/iw_data_pipeline/services/dbt_controller.py", line 172, in trigger_dbt_job trigger_response = await admin_client.trigger_job_run( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect_dbt/cloud/clients.py", line 150, in trigger_job_run return await self.call_endpoint( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect_dbt/cloud/clients.py", line 56, in call_endpoint response = await self._admin_client.request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1540, in request return await self.send(request, auth=auth, follow_redirects=follow_redirects) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1629, in send response = await self._send_handling_auth( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1657, in _send_handling_auth response = await self._send_handling_redirects( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects response = await self._send_single_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1730, in _send_single_request response = await transport.handle_async_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 393, in handle_async_request with map_httpcore_exceptions(): File "/usr/local/lib/python3.11/contextlib.py", line 158, in exit self.gen.throw(typ, value, traceback) File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ConnectError: [Errno -2] Name or service not known
m
✅ thought for 150.5 seconds
Thanks for sharing the traceback — that httpx.ConnectError “[Errno -2] Name or service not known” indicates a DNS resolution failure for the host the dbt Cloud client is trying to reach. In Prefect 3.x, the dbt Cloud admin client builds requests against: https://{domain}/api/v2/accounts/{account_id} So if the 
domain
is wrong or cannot be resolved from where your task is running, you’ll see exactly this error. Quick checks - What environment are you running this task in? (local, Docker, Kubernetes, behind a corporate proxy?) - How are you creating your 
DbtCloudCredentials
? Specifically, what value is set for 
domain
and 
account_id
? (Please redact your API key.) - Can the environment resolve and reach the dbt Cloud domain (e.g., cloud.getdbt.com or your region/custom domain)? Common causes and fixes 1) Domain is incorrect or includes a scheme - The 
domain
must be a hostname only (no https://). Examples: - Correct: 
<http://cloud.getdbt.com|cloud.getdbt.com>
- Correct (regional/custom): 
<http://eu.cloud.getdbt.com|eu.cloud.getdbt.com>
or 
<http://dbt.company.com|dbt.company.com>
- Incorrect: 
<https://cloud.getdbt.com>
- If you are on a regional or enterprise instance, use the exact host you see when you log into dbt Cloud. 2) DNS cannot resolve the host from within your worker/container - From the same runtime where the flow runs, test:
Copy code
nslookup <http://cloud.getdbt.com|cloud.getdbt.com>
  curl -I <https://cloud.getdbt.com>
If these fail, fix DNS or egress/network policy for that environment. 3) Proxy/firewall egress restrictions - If you require a proxy, set standard env vars before starting your worker/flow:
Copy code
export HTTPS_PROXY="<http://proxy.company.com:8080>"
  export HTTP_PROXY="<http://proxy.company.com:8080>"
  export NO_PROXY="localhost,127.0.0.1"
httpx (used by prefect-dbt) will pick these up automatically. 4) Typo in regional/custom domain - A small typo (e.g., 
<http://cloud.getdbt.co|cloud.getdbt.co>
) will cause this exact error. How the URL is built (to help you verify) - Prefect’s dbt Cloud admin client uses an httpx.AsyncClient with: - base_url: 
https://{domain}/api/v2/accounts/{account_id}
- So a quick way to sanity check is to log/print what your credentials resolve to. Minimal verification code Run this inside your flow’s environment to confirm what domain is set and whether it resolves:
Copy code
from prefect import flow, task
from prefect_dbt.cloud import DbtCloudCredentials
import socket

@task
def inspect_dbt_cloud_host(creds: DbtCloudCredentials):
    print(f"dbt Cloud domain: {creds.domain}")
    print(f"dbt Cloud account_id: {creds.account_id}")
    try:
        ip = socket.gethostbyname(creds.domain)
        print(f"Resolved {creds.domain} to {ip}")
    except Exception as e:
        print(f"DNS resolution failed for {creds.domain}: {e}")
        raise

@flow
def diagnose():
    # If you’re loading from a saved block, replace this with .load("my-block")
    creds = DbtCloudCredentials(
        api_key="REDACTED",
        account_id=123456789,
        # domain="<http://cloud.getdbt.com|cloud.getdbt.com>",  # or your regional/custom domain
    )
    inspect_dbt_cloud_host(creds)

if __name__ == "__main__":
    diagnose()
Example: creating the credentials correctly
Copy code
from prefect_dbt.cloud import DbtCloudCredentials

# Standard multi-tenant (default)
creds = DbtCloudCredentials(
    api_key="REDACTED",
    account_id=123456789,
    # domain defaults to "<http://cloud.getdbt.com|cloud.getdbt.com>"
)

# Regional/custom (enterprise)
creds = DbtCloudCredentials(
    api_key="REDACTED",
    account_id=123456789,
    domain="<http://eu.cloud.getdbt.com|eu.cloud.getdbt.com>"  # or your exact enterprise hostname
)
If you share: - The value you’re using for 
domain
(no secrets), and - Where the flow is running (local/Docker/K8s/proxy), I can help pinpoint the issue and suggest the exact fix for your environment.
5 Views
Open in Slack
PreviousNext
Powered by