Short answer: you can’t “reference” an existing Job object in the cluster via a job variable. In Prefect 3.x, Kubernetes workers create Jobs from a base job template on the work pool plus per-deployment job_variables. The supported pattern is to turn your existing Job manifest into the work pool’s base job template (with Jinja placeholders for anything you want to vary), then pass small overrides in job_variables at deploy time. Recommended approach 1) Export the default template and use it as a starting point

prefect work-pool get-default-base-job-template --type kubernetes -f base-job-template.json

2) Replace the job spec with your existing manifest - Paste your full Job manifest into the template’s job section. - Keep or add placeholders for anything you want to vary per deployment (e.g., image, env, namespace, labels, serviceAccountName). - You can hardcode values you never want to change. 3) Update your work pool with the custom template

prefect work-pool update <your-pool> --base-job-template base-job-template.json

(You can also do this in the UI under Work Pools > your pool > Base job template.) 4) Provide job_variables on your deployments - In your

prefect.yaml

or in code via

flow.from_source(...).deploy(job_variables={...})

, set only the values that correspond to placeholders you left in the template. Notes and tips - You cannot point to an existing Job by name; the worker needs a manifest to create a new Job per flow run. - If you really want to pass the entire manifest per-deployment, you can design a base job template that defers most of the spec to a single object variable (e.g., define a

variables.job

of type object and use

{{ job }}

in the template). However, the more common and maintainable pattern is to set the manifest as the base job template and override just a few fields via job_variables. - To see what variables are available or expected, inspect the default template you exported; any placeholders you keep are the ones you can override at deploy time. Docs - Manage work pools (base job templates) - Customize job variables - Workers overview If you can share a redacted version of your Job manifest and which fields you want to change per deployment (image, env, namespace, resources, etc.), I can help you turn it into a base job template and map the corresponding job_variables.

Great question! Where to put prefect.yaml - Create it at the root of your project (the directory you’ll run commands from and where your flow code lives). The Prefect CLI looks for

prefect.yaml

in the current working directory. How to create it - Quick scaffold:

prefect init

- Or create it manually. Example for a Kubernetes work pool:

name: my-project

deployments:
  - name: my-flow-prod
    entrypoint: flows/my_flow.py:my_flow
    tags: ["prod"]

    work_pool:
      name: my-k8s-pool

    # These map to placeholders in your work pool's base job template
    job_variables:
      image: <http://ghcr.io/my-org/my-image:sha-123|ghcr.io/my-org/my-image:sha-123>
      namespace: my-namespace
      service_account_name: my-service-account
      env:
        - name: PREFECT_LOGGING_LEVEL
          value: INFO

    parameters:
      x: 1

    schedules:
      - cron: "0 * * * *"
        timezone: UTC

How to “sync” it to Prefect (register deployments) 1) Make sure your CLI points to your self-hosted API:

prefect config set PREFECT_API_URL=https://<your-prefect-server>/api

2) From the same directory as

prefect.yaml

, run: - Deploy all entries in the file:

prefect deploy --all

- Deploy a specific one:

prefect deploy --name my-flow-prod

This pushes/updates the deployment(s) on your Prefect server. Re-run

prefect deploy

whenever you change the file. Tips - You can keep your base Kubernetes Job manifest on the work pool (Base job template) and only set small overrides via

job_variables

in

prefect.yaml

. - You can also pass an

ENTRYPOINT

directly to

prefect deploy

instead of using

prefect.yaml

, but

prefect.yaml

is best for repeatable, versioned deployments. Docs - prefect.yaml reference - Manage work pools (base job templates) - Customize job variables If you share your repo layout and the fields you want to vary (image, env, namespace, etc.), I can tailor a

prefect.yaml

and the matching work pool template for you.

Short answer: not directly. Argo CD manages Kubernetes manifests, while

prefect.yaml

is a Prefect project config consumed by the Prefect CLI. To “sync” it with Argo CD, wrap the

prefect deploy

step in a Kubernetes object that Argo CD can apply (commonly a Job run as a sync hook). Two common patterns 1) Use CI to register deployments (recommended) - Run

prefect deploy --all

in your CI (GitHub Actions, GitLab, etc.) whenever

prefect.yaml

or flow code changes. - Let Argo CD manage only your Kubernetes infra (work pool, base job template, workers, etc.). - Pros: simpler permissions and fewer moving parts in-cluster. 2) Register via Argo CD PostSync hook (in-cluster) - Create a Kubernetes Job that runs

prefect deploy --all

after Argo CD syncs the repo. - Ensure the Job has: - Access to your project code and

prefect.yaml

(bake them into the image or git-clone in the Job using a secret). - Prefect CLI available (preinstalled in the image or install at runtime). -

PREFECT_API_URL

set to your self-hosted Prefect server API. - Example hook Job manifest you can include in your Argo CD app:

apiVersion: batch/v1
kind: Job
metadata:
  name: prefect-deploy
  annotations:
    <http://argocd.argoproj.io/hook|argocd.argoproj.io/hook>: PostSync
    <http://argocd.argoproj.io/hook-delete-policy|argocd.argoproj.io/hook-delete-policy>: HookSucceeded
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: deploy
          # Option A: custom image with your repo + Prefect CLI inside
          image: <http://ghcr.io/your-org/prefect-project:latest|ghcr.io/your-org/prefect-project:latest>
          env:
            - name: PREFECT_API_URL
              value: <https://your-prefect-server.example.com/api>
          command: ["/bin/sh","-lc"]
          args:
            - |
              prefect deploy --all
          # Option B: fetch code at runtime (private repo requires credentials)
          # image: python:3.11-slim
          # command: ["/bin/sh","-lc"]
          # args: |
          #   pip install "prefect==3.4.20"
          #   git clone <https://github.com/your/repo.git> /workspace
          #   cd /workspace
          #   prefect deploy --all

Notes -

prefect deploy

is idempotent; it updates existing deployments. - If your work pool/base job template is also GitOps-managed, you can use Argo CD sync waves or hook ordering to ensure the pool/template is applied before the deploy Job runs. - Make sure your Job’s service account has network access to your Prefect API and any needed secrets for repo access. - For Prefect 3.x,

prefect deploy

is the correct command (do not use the removed 2.x commands). References - Prefect

prefect.yaml

and deploy: https://docs-3.prefect.io/v3/how-to-guides/deployments/prefect-yaml - Work pools and base job templates: https://docs-3.prefect.io/v3/how-to-guides/deployment_infra/manage-work-pools - Customizing job variables: https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables If you share how your repo is structured (and whether you can build a small “registration” image), I can sketch the exact Job manifest and hook setup to fit your environment.

Yes—you can bake your flow code into a custom image and have Kubernetes workers run it. The key is: - The worker image (prefect-kubernetes) just launches Jobs. - The Job’s container image should be your custom image that already contains the flow code. - The deployment’s

entrypoint

must point to the file path inside your image. Here’s a minimal, working setup. 1) Ensure your work pool’s base job template uses a templated image - In your Kubernetes work pool, the Base job template should have a placeholder like

{{ image }}

for the container image. That way, deployments can set

job_variables.image

per deployment. - You can export the default and edit:

prefect work-pool get-default-base-job-template --type kubernetes -f base-job-template.json
prefect work-pool update my-k8s-pool --base-job-template base-job-template.json

2) Create a prefect.yaml at your project root that points to the path inside your image - Put the flow code into your custom image at a known path (e.g.,

/opt/flows/my_flow.py

), and make sure Prefect and dependencies are installed in that image. - Then configure your deployment to use that path and your image, and disable code pulling because the code is already in the image:

name: my-project

deployments:
  - name: my-flow
    entrypoint: /opt/flows/my_flow.py:my_flow  # path inside your image
    tags: ["prod"]
    work_pool:
      name: my-k8s-pool
    job_variables:
      image: <http://ghcr.io/your-org/your-flows:sha-123|ghcr.io/your-org/your-flows:sha-123>
    # No runtime code fetching; code is already in the image
    pull: []

3) Register the deployment with your Prefect server - Make sure your CLI points at your self-hosted API:

prefect config set PREFECT_API_URL=https://<your-prefect-server>/api

- From the directory with

prefect.yaml

, run:

prefect deploy --name my-flow

(or

prefect deploy --all

to register all entries in the file) That’s it. When a flow run is picked up: - The Kubernetes worker will render your base job template. - It will set the job’s container image to

<http://ghcr.io/your-org/your-flows:sha-123|ghcr.io/your-org/your-flows:sha-123>

. - The container will start, and Prefect will execute the entrypoint

/opt/flows/my_flow.py:my_flow

that’s already in the image. Tips - The

entrypoint

path must exist inside your image. Use an absolute path to avoid working directory surprises. - Keep Prefect versions compatible between your image and server (you mentioned 3.4.20—pin that in your image). - If you want to vary only the image tag per environment, keep the

{{ image }}

placeholder in the work pool’s base template and set it via

job_variables

per-deployment or environment. Docs - prefect.yaml reference: https://docs-3.prefect.io/v3/how-to-guides/deployments/prefect-yaml - Manage work pools/base job template: https://docs-3.prefect.io/v3/how-to-guides/deployment_infra/manage-work-pools - Customize job variables: https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables If you can share the path where your flow file lives in the image and the name of your work pool, I can tailor the

prefect.yaml

and confirm the exact base template placeholders to use.

That error happens because

prefect deploy

tries to resolve/import the entrypoint on the machine where you run the command. Your entrypoint is an absolute path inside your container (

/app/...

), but that file doesn’t exist on your laptop, so Prefect raises

FileNotFoundError

. You’ve got a few workable options: Option A (recommended): Use a module path for the entrypoint and make it importable locally - Change

entrypoint

to a module import instead of a filesystem path, e.g.:

entrypoint: equall.services.agents.src.pipelines.dummy:dummy_flow

- Ensure your project is importable in the environment where you run `prefect deploy`: - Either

pip install -e .

from your repo root, or set

PYTHONPATH

so

equall/...

is importable. - Make sure the same module path is importable inside your container image (e.g., install the package during the image build or set

PYTHONPATH=/app

). Quick local check:

python -c "from equall.services.agents.src.pipelines.dummy import dummy_flow; print(dummy_flow)"

Option B: Run

prefect deploy

inside the same image that contains your code - Execute the deploy from a container where

/app/equall/...

exists:

docker run --rm \
  -e PREFECT_API_URL=https://<your-prefect-server>/api \
  -v $PWD/prefect.yaml:/app/prefect.yaml \
  <http://ghcr.io/your-org/your-flows:sha-123|ghcr.io/your-org/your-flows:sha-123> \
  sh -lc 'prefect deploy --name my-flow'

- This way, the entrypoint

/app/equall/...

exists at deploy time. Option C: Use a local path that exists, but keep runtime code in the image - If you have the repo locally, you can set:

entrypoint: ./services/agents/src/pipelines/dummy.py:dummy_flow
pull: []

- Prefect will resolve the file locally for registration; at runtime the container executes the same entrypoint string. If your container’s working directory differs, this can be brittle—module paths are more robust. Why this is needed - Even when you set

pull: []

and run with a custom image, Prefect still needs to import/resolve the flow at deploy time to register the deployment. That resolution happens where you run

prefect deploy

. Your current file (with a module path) would look like:

name: equall

deployments:
  - name: my-flow
    entrypoint: equall.services.agents.src.pipelines.dummy:dummy_flow
    tags: ["test"]
    work_pool:
      name: dataroom
    job_variables: {}
    pull: []

Then run:

prefect config set PREFECT_API_URL=https://<your-prefect-server>/api
prefect deploy --name my-flow

Docs - prefect.yaml reference - Customize job variables If you share how your code is laid out in the image (paths, PYTHONPATH, working dir), I can suggest the exact entrypoint and Dockerfile tweaks to make the module path work seamlessly both locally and in-cluster.

That “Expecting value: line 1 column 1 (char 0)” is a JSONDecodeError. In practice, during

prefect deploy

it almost always comes from one of these: 1) PREFECT_API_URL is wrong (points to the UI, not the API) - If your CLI is hitting an HTML page (e.g., https://your-prefect.example.com) instead of the API (https://your-prefect.example.com/api), the client tries to parse HTML as JSON and fails at char 0. - Verify:

prefect config view --show-sources

Ensure

PREFECT_API_URL

ends with

/api

and is reachable:

curl -sS -i <https://your-prefect.example.com/api/health>

You should see a 200 and JSON body. If you see HTML or a redirect, fix the URL or auth. 2) Invalid base job template on the work pool (YAML uploaded where JSON was expected, or malformed JSON) - If your work pool’s Base job template was uploaded from a YAML file or contains comments/trailing commas, Prefect may hit a JSON parse error when reading it. - Inspect it:

prefect work-pool inspect dataroom

Look for

base_job_template

. If it’s empty, odd, or not valid JSON, re-export and re-apply:

prefect work-pool get-default-base-job-template --type kubernetes -f base-job-template.json
# edit as needed (keep valid JSON; use Jinja placeholders like {{ image }})
prefect work-pool update dataroom --base-job-template base-job-template.json

3) The entrypoint or project import fails and some handler masks the root exception - Confirm you can import the flow locally the same way

prefect deploy

will:

python -c "from equall.services.agents.src.pipelines.dummy import dummy_flow; print(dummy_flow)"

If that fails, fix your local PYTHONPATH or install your package locally (e.g.,

pip install -e .

). 4) Corrupted or invalid prefect.yaml (less common) - Quick validation:

python - <<'PY'
import sys, yaml
with open("prefect.yaml") as f:
    yaml.safe_load(f)
print("prefect.yaml OK")
PY

If the API URL is correct and the base job template is valid JSON, this error usually disappears. Given your setup: - You’re running

prefect deploy

from your laptop. - Your work pool image is set in the Base job template already. - Your deployment uses a module path or a local path that exists. Do this sequence: 1) Verify API URL and health:

prefect config view --show-sources
curl -sS -i $PREFECT_API_URL/health

2) Inspect work pool template:

prefect work-pool inspect dataroom

If needed, reapply a valid JSON base job template. 3) Validate local import:

python -c "from equall.services.agents.src.pipelines.dummy import dummy_flow; print(dummy_flow)"

4) Re-run:

prefect deploy --name my-flow

If it still fails, please paste the full stack trace from

prefect deploy

and the output of: -

prefect config view --show-sources

-

prefect work-pool inspect dataroom

(redact secrets) - Your current

prefect.yaml

Docs for reference: - prefect.yaml reference: https://docs-3.prefect.io/v3/how-to-guides/deployments/prefect-yaml - Manage work pools / base job template: https://docs-3.prefect.io/v3/how-to-guides/deployment_infra/manage-work-pools

Good catch. That entry you’re seeing is Prefect’s “set working directory” pull step. When you run prefect deploy locally, Prefect infers a project working directory from where it found your flow/where you ran the command and records it as a pull step so that, at runtime, the worker will chdir into that directory before importing your flow. Since you deployed from your laptop, it captured your local path. With code baked into a container image, you don’t want a macOS path in there. You have two options: - Remove the step entirely - Override it to a path that exists inside your image (e.g., /app) Examples you can use in your prefect.yaml: - Remove it

deployments:
  - name: my-flow
    entrypoint: equall.services.agents.src.pipelines.dummy:dummy_flow
    work_pool:
      name: dataroom
    job_variables: {}
    pull: []

- Override to a container path

deployments:
  - name: my-flow
    entrypoint: equall.services.agents.src.pipelines.dummy:dummy_flow
    work_pool:
      name: dataroom
    job_variables: {}
    pull:
      - prefect.deployments.steps.set_working_directory:
          directory: /app

Then re-register:

prefect deploy --name my-flow

Tips - Make sure the path you set (e.g., /app) exists in your image and matches where your code lives. - Verify what’s stored on the server:

prefect deployment inspect "dummy_flow/my-flow"

- If you still see the local path after setting

pull: []

, double-check you’re deploying the same entry from prefect.yaml and that the UI is showing the latest version. Docs - Pull steps (including set_working_directory): https://docs-3.prefect.io/v3/how-to-guides/deployments/prefect-yaml (see pull steps) - API reference for pull steps: https://docs-3.prefect.io/v3/api-ref/python/prefect-deployments-steps-pull

Absolutely — if you share your prefect.yaml, I can translate it 1:1. In the meantime, here’s the general Python pattern to deploy the same flow in Prefect 3.x using Flow.from_source(...).deploy(...). Example: deploy a locally defined dummy_flow from your project folder

from prefect import Flow
from prefect.schedules import Cron

# This assumes your flow is defined at flows/dummy.py and is named `dummy_flow`
deployment_id = Flow.from_source(
    source=".",  # project root (can be a local path)
    entrypoint="flows/dummy.py:dummy_flow",  # "<relative_path_to_file>:<flow_function_name>"
).deploy(
    name="dummy-deployment",                 # maps to `name` in prefect.yaml
    work_pool_name="my-work-pool",           # maps to `work_pool.name`
    # work_queue_name="default",             # maps to `work_pool.queue` if you used one
    parameters={"param": 1},                 # maps to `parameters`
    tags=["dev", "demo"],                    # maps to `tags`
    description="Dummy flow deployed via Python",
    # single schedule:
    schedule=Cron("0 * * * *", timezone="UTC"),  # maps to a cron schedule
    # or multiple schedules:
    # schedules=[Cron("0 9 * * MON-FRI", timezone="UTC")],
    # If your pool runs containers, you can set an image:
    # image="<http://ghcr.io/org/project:latest|ghcr.io/org/project:latest>",
    # Or pass pool-specific job variables (overrides):
    job_variables={
        "env": {"PREFECT_LOGGING_LEVEL": "DEBUG"},
        # For Docker/K8s pools you can also put image here, e.g.:
        # "image": "<http://ghcr.io/org/project:latest|ghcr.io/org/project:latest>"
    },
)

print("Created deployment:", deployment_id)

Example: deploy the same flow from a GitHub repo

from prefect import Flow

deployment_id = Flow.from_source(
    source="<https://github.com/owner/repo>",  # repo URL
    entrypoint="flows/dummy.py:dummy_flow",  # file + function in that repo
).deploy(
    name="dummy-deployment",
    work_pool_name="my-work-pool",
    parameters={"param": 1},
    tags=["dev", "demo"],
)

If your prefect.yaml used a Docker image and build/push settings, you can mirror that too:

deployment_id = Flow.from_source(
    source=".",
    entrypoint="flows/dummy.py:dummy_flow",
).deploy(
    name="dummy-docker",
    work_pool_name="docker-pool",
    image="<http://ghcr.io/org/app:latest|ghcr.io/org/app:latest>",  # requires a container-based pool
    build=True,   # build image from your Dockerfile
    push=True,    # push image to the registry
)

Mapping from prefect.yaml fields to code (common ones) - name -> deploy(name=...) - entrypoint -> Flow.from_source(..., entrypoint="path:flow_func") - work_pool.name -> deploy(work_pool_name=...) - work_pool.queue -> deploy(work_queue_name=...) - parameters -> deploy(parameters={...}) - tags -> deploy(tags=[...]) - description -> deploy(description="...") - schedules (cron/interval/rrule) -> deploy(schedule=...) or deploy(schedules=[...]) - image -> deploy(image="...") or deploy(job_variables={"image": "..."}) - job_variables -> deploy(job_variables={...}) - version/tags -> deploy(version=..., tags=[...]) Docs for reference: - Deploy via Python: Deploy via Python - Job variables (infra overrides): Customize job variables If you paste your prefect.yaml here, I’ll give you the exact equivalent Python snippet for your dummy_flow.

Great question. In your prefect.yaml the pull step

pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /app

just sets the working directory on the run environment; it doesn’t actually pull code. In Python, you mirror this via worker-specific job variables. If you want to ensure the worker does not pull any code at all, do NOT use Flow.from_source(...). Instead, deploy from the imported flow object so the deployment has no storage attached and the worker will import your code that is already present on the machine/container. Option A — no code pull (recommended if your workers already have the code at /app)

from equall.services.agents.src.pipelines.dummy import dummy_flow

deployment_id = dummy_flow.deploy(
    name="my-flow",
    work_pool_name="dataroom",
    # mirrors the YAML pull step
    job_variables={
        "working_dir": "/app"  # Process/Docker workers support this
    },
    # if using a container-based pool, ensure your image already contains the code
    # image="<http://ghcr.io/your-org/your-image:tag|ghcr.io/your-org/your-image:tag>",  # when using Docker/K8s pools
)
print(deployment_id)

Option B — still package code with the deployment (will “pull” the packaged code at runtime) Only use this if you’re okay with the worker fetching the deployment’s storage bundle:

from prefect import Flow

deployment_id = Flow.from_source(
    source=".",  # creates/attaches deployment storage; will be fetched by the worker
    entrypoint="equall.services.agents.src.pipelines.dummy:dummy_flow",
).deploy(
    name="my-flow",
    work_pool_name="dataroom",
    job_variables={"working_dir": "/app"},
)

Notes - Process worker: use job_variables={"working_dir": "/app"}. - Docker worker: same key works; also set an image if needed, and make sure the image has your code baked in to avoid any external pull. - Kubernetes worker: set the container workingDir in the pod spec via job_variables; or bake code into the image. Docs - Deploy via Python: Deploy via Python - Job variables: Customize job variables Quick check: what type is your “dataroom” work pool (process, docker, kubernetes)? I can tailor the exact job_variables for it.