https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> after I created a work pool of type `...
# ask-marvin
m
@Marvin after I created a work pool of type `prefect:managed`in Prefect Cloud using Terraform, how do I use it to run dockerized jobs?
m
✅ thought for 143.9 seconds
If your work pool is 
type = "prefect:managed"
, the key thing to know is: You do not use it to run your own custom Docker image. A 
prefect:managed
pool runs on Prefect-managed infrastructure using an official Prefect base image. You can point deployments at that pool, and Prefect Cloud will execute them for you, but you can’t supply a custom Docker image the way you would with a Docker, Kubernetes, ECS, or Cloud Run style pool. So the answer depends on what you mean by “dockerized jobs”: * If you mean “run my flow in a containerized environment managed by Prefect” → yes, use the managed pool. * If you mean “run my own custom Docker image” → no, 
prefect:managed
is not the right pool type. Docs: - Managed work pools - Work pools concepts - Serverless / push-style infrastructure How to use your 
prefect:managed
pool You create a deployment that targets that pool. Example with Python:
Copy code
python
from prefect import flow

@flow
def my_flow():
    print("hello from Prefect-managed infrastructure")


if __name__ == "__main__":
    my_flow.from_source(
        source="<https://github.com/your-org/your-repo.git>",
        entrypoint="flows/my_flow.py:my_flow",
    ).deploy(
        name="managed-deployment",
        work_pool_name="your-managed-pool",
        job_variables={
            "pip_packages": ["pandas", "requests"]
        },
    )
A few important notes: - Your code must be available from remote storage, like Git or blob/object storage. - For extra Python dependencies, use 
job_variables={"pip_packages": [...]}
. - Prefect Cloud handles the infrastructure submission automatically; you do not run a worker for 
prefect:managed
. You can also do this with 
prefect.yaml
and then run 
prefect deploy
. Verified CLI:
Copy code
bash
prefect deploy --help
Example `prefect.yaml`:
Copy code
yaml
deployments:
  - name: managed-deployment
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: your-managed-pool
      job_variables:
        pip_packages:
          - pandas
          - requests
Then:
Copy code
bash
prefect deploy --prefect-file prefect.yaml
If you need a custom Docker image Then you should use a different work pool type, for example: - 
docker
work pool - 
kubernetes
work pool - 
ecs
/ 
ecs:push
- 
cloud-run:push
- 
azure-container-instance
/ push variants Those are the options that support bringing your own image. So in short: - 
prefect:managed
→ managed container execution, but not your custom image - custom Docker image → use another pool type If you want, I can show you the exact setup for either: 1. deploying to your existing 
prefect:managed
pool, or 2. switching to a work pool type that runs your own Docker image
m
@Marvin I want to bring my own Docker image. I changed the type to 
docker
in my Terraform file, and when trying to 
terraform apply
, got this error: "Your plan does not support hybrid or push work pools." Does this mean I cannot do this on Prefect Cloud's free tier?
m
✅ thought for 11.0 seconds
Yes — that error generally means your current Prefect Cloud plan does not support 
docker
work pools. A 
docker
work pool is a hybrid work pool, and the message:
Copy code
text
Your plan does not support hybrid or push work pools.
means your workspace’s plan only allows 
prefect:managed
pools, not hybrid/push infrastructure. So on the free tier / lower-tier plans, you typically cannot bring your own Docker image through a 
docker
work pool. Relevant links: - Prefect pricing - Prefect Cloud - Work pools concepts What your options are 1. Stay on 
prefect:managed
 - No worker to run - No custom image support - Can install Python packages at runtime with 
pip_packages
2. Upgrade to a plan that supports hybrid/push work pools - Self-serve path: Org Settings → Billing → Upgrade in Prefect Cloud - Pricing details: prefect.io/pricing 3. If you need something beyond self-serve plan changes - Use Prefect contact If you want, I can help you figure out whether your image requirements can be handled with 
prefect:managed
+ 
pip_packages
, or whether you truly need a hybrid/push pool.
m
@Marvin in that case how do you ensure parity between the environment running locally and the one managed by Prefect Cloud?
m
✅ thought for 163.3 seconds
The short answer is: you don’t get perfect image-level parity with 
prefect:managed
, so you aim for Python/runtime parity instead. Since you can’t bring your own Docker image, the best practice is to make your flow environment reproducible through: 1. pinning Python versions 2. pinning Python dependencies 3. storing code remotely and deploying from source 4. testing in a clean environment that mimics managed execution Docs: - Managed work pools - Deploy via Python - Store flow code - Test workflows Recommended approach 1. Pin your Python version locally Use the same minor Python version locally that you expect in cloud execution, for example 
3.12
. Examples: - 
pyproject.toml
- 
uv
- 
pyenv
- CI config The main idea is: don’t develop on 
3.13
and assume it will behave the same as 
3.11
. 2. Pin all Python dependencies exactly This is the most important part. Use a locked 
requirements.txt
or equivalent, for example:
Copy code
text
prefect==3.0.0
pandas==2.2.2
requests==2.32.3
prefect-aws==0.5.5
Then install those same dependencies in managed execution. For a managed pool, that usually means either: - 
job_variables.pip_packages
, or - a pull step that installs from 
requirements.txt
Example with `prefect.yaml`:
Copy code
yaml
pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://github.com/your-org/your-repo.git>
      branch: main
  - prefect.deployments.steps.pip_install_requirements:
      requirements_file: requirements.txt
      directory: "{{ clone-step.directory }}"

deployments:
  - name: my-deployment
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-managed-pool
That gets you much closer to parity than manually installing packages ad hoc. 3. Treat your repo as the source of truth For managed pools, your flow code should live in Git or remote storage, and deployments should use that source directly. That way: - local runs use the checked-out repo - cloud runs pull the same repo/ref - you can pin to a branch, commit, or tag Example:
Copy code
python
from prefect import flow

@flow
def my_flow():
    print("hello")

if __name__ == "__main__":
    my_flow.from_source(
        source="<https://github.com/your-org/your-repo.git>",
        entrypoint="flows/my_flow.py:my_flow",
    ).deploy(
        name="prod",
        work_pool_name="my-managed-pool",
        job_variables={
            "pip_packages": [
                "prefect==3.0.0",
                "pandas==2.2.2",
            ]
        },
    )
4. Test in a clean environment locally A very practical pattern is to test your flow in a fresh virtualenv or a simple local Docker container based on 
python:X.Y-slim
, then install only your pinned dependencies. That won’t be identical to Prefect-managed infra, but it catches most parity problems: - missing package pins - transitive dependency drift - missing imports - assumptions about local machine state For example:
Copy code
bash
python -m venv /tmp/prefect-parity
source /tmp/prefect-parity/bin/activate
pip install -r requirements.txt
python flows/my_flow.py
5. Avoid relying on OS-level/system packages if possible This is one of the biggest limitations of 
prefect:managed
. If your flow depends on: - OS libraries - custom binaries - apt-installed packages - special drivers - GPU tooling then parity becomes hard or impossible with managed pools. If you can, prefer pure-Python packages like: - 
psycopg2-binary
instead of compiling 
psycopg2
- wheels that don’t require system build tools If you must control system-level dependencies, that’s usually the point where you need a custom image, which means a hybrid/push pool on a plan that supports it.
6. Put configuration in env vars / blocks, not local machine state Don’t let local parity depend on: - files in your home directory - local AWS credentials silently being present - locally mounted paths - developer-specific shell config Instead use: - Prefect blocks / secrets - environment variables - remote storage - cloud auth patterns like workload identity where possible That way local and cloud both resolve config explicitly. Reality check / limitation With 
prefect:managed
, you can get very good parity for Python code and dependencies, but not full container parity. So think of it like this: - Good fit: standard Python ETL, APIs, orchestration, db access, cloud SDKs - Poor fit: custom OS packages, native binaries, GPU jobs, highly specialized runtime requirements My practical recommendation If you must stay on 
prefect:managed
, I’d do this: 1. Pin Python minor version 2. Generate a locked 
requirements.txt
3. Store flows in Git 4. Deploy with 
from_source(...)
5. Install dependencies from that lock file in the deployment 6. Run CI tests in a fresh environment before deploying If you want, I can help you build a concrete managed-pool deployment setup with: - 
pyproject.toml
- 
requirements.txt
- 
prefect.yaml
- CI example so you have a repeatable parity workflow end to end.
m
@Marvin That sounds like bad Software Engineering practice. There's a reason why Docker is so popular. I'd rather you gave the simple answer "for your use case, indeed you have to use one of the paid tiers".
m
✅ thought for 5.0 seconds
That’s fair — and yes, the simple answer is: For your use case, if you need to bring your own Docker image, you need a paid Prefect Cloud tier that supports hybrid or push work pools. On the free tier, you’re effectively limited to 
prefect:managed
, which does not support custom images. Relevant links: - Prefect pricing - Prefect Cloud billing - Work pools concepts So the practical decision is: - Need custom Docker image → upgrade to a plan with hybrid/push work pools - Staying on free tier → use 
prefect:managed
and accept its environment constraints Your criticism is reasonable; if strict container parity is a requirement, 
prefect:managed
is not the right fit.
m
@Marvin Thank you. The cheapest tier is 100$ / month, which is way too much for a hobby project. That effectively rules out Prefect Cloud as an option, I'll look into cheaper alternatives. You might want to pass this information along to whomever collects customer / sales feedback.
m
✅ thought for 20.8 seconds
Totally understandable. And just to be direct: if your hobby project requires bring-your-own Docker images, Prefect Cloud is probably not a good fit at that price point. I also want to be transparent that I can’t forward feedback internally or pass it to sales/product on your behalf. If you want to share that feedback directly with Prefect, the public path is: - Prefect contact If it helps, there is one lower-cost path that may still keep Prefect in play: - use open-source Prefect and run it yourself, rather than Prefect Cloud That gives you much more freedom around Dockerized execution, since you’re managing the infrastructure yourself. Docs: - Prefect getting started - Work pools concepts If you want, I can help you sketch the cheapest self-hosted Prefect setup for a hobby project using Docker on a single VM or laptop.
m
@Marvin If I use that self-hosted option, and only having a single laptop, how can I ensure that data on the server (such as logs of flows, etc) persists even when I shut down the laptop?
m
✅ thought for 136.6 seconds
Yes — the metadata persists, as long as you store it on disk. But there’s an important distinction: - Persistence of data → yes, you can keep flow history, logs, state, deployments, etc. - Continued operation while the laptop is off → no, nothing runs while the laptop is shut down So if your laptop is off: - the Prefect server is down - the UI is unavailable - the scheduler is not running - workers are not polling - scheduled runs due during that time generally won’t execute But when you turn it back on, the stored data can still be there. Docs: - Prefect getting started - Work pools concepts For a single-laptop setup, the simplest answer is: Use self-hosted Prefect with persistent local storage. By default, Prefect server uses a local SQLite database, which is typically stored under:
Copy code
text
~/.prefect/prefect.db
That database holds things like: - flow/task run metadata - state history - deployments - logs/records stored in the API DB - blocks, variables, artifacts, etc. So if that file remains on disk, your history survives reboots. If you run Prefect directly on the laptop Then persistence is basically automatic, because the database is just a local file. Example:
Copy code
bash
prefect server start
When you stop the server and shut down the laptop, 
~/.prefect/prefect.db
remains on disk. When you restart later:
Copy code
bash
prefect server start
your previous metadata should still be there. If you run Prefect in Docker Then you must mount a volume, or the container’s database will disappear when the container is removed. Example:
Copy code
bash
docker run \
  -p 4200:4200 \
  -v ~/.prefect:/root/.prefect \
  prefecthq/prefect:3-latest \
  prefect server start --host 0.0.0.0
That 
-v ~/.prefect:/root/.prefect
is the important part. It ensures the SQLite DB and other local Prefect data live on your laptop filesystem, not only inside the container. A simple 
docker-compose.yml
version:
Copy code
yaml
services:
  prefect:
    image: prefecthq/prefect:3-latest
    ports:
      - "4200:4200"
    volumes:
      - ~/.prefect:/root/.prefect
    command: prefect server start --host 0.0.0.0
What will not persist if the laptop is off? Even with persistent storage, the laptop being off means: - no scheduled jobs will run during downtime - no API/UI access - no automations firing - no workers doing work So persistence solves data retention, not availability. If you need both: - persistent metadata and - workflows still running while your laptop is off then you need something that stays on: - a small VPS - a home server / Raspberry Pi - a cloud VM - or Prefect Cloud Best hobby-project recommendation For the cheapest workable setup: - run Prefect server on your laptop - use the default SQLite DB - if using Docker, mount 
~/.prefect
- optionally back up 
~/.prefect/prefect.db
That gives you persistent history/logs/state with almost no cost. If you want more reliability The next step up is: - host Prefect on a tiny always-on VPS - use Docker Compose - mount a persistent volume - optionally use Postgres instead of SQLite That way both the server data and the server availability survive your laptop being off. So the concise answer is: Yes, the server data can persist across laptop shutdowns, but only if it is stored on persistent disk (for example SQLite in 
~/.prefect
or a Docker-mounted volume). However, nothing continues running while the laptop is off. If you want, I can give you a minimal self-hosted hobby setup for either: 1. single laptop only, or 2. cheap always-on VPS with Docker Compose.
m
@Marvin Let's do the single laptop version just for my curiosity. Though in the future I'll want a managed version.
m
✅ thought for 23.7 seconds
Sure — here’s the minimal single-laptop self-hosted Prefect setup. What you get - Prefect API/UI running locally - metadata persisted on your laptop - ability to create deployments and run flows - zero cloud cost What you do not get - runs while the laptop is off - always-on scheduling - managed infrastructure Option A: simplest possible setup (no Docker) 1. Install Prefect
Copy code
bash
pip install "prefect>=3,<4"
2. Start the local server
Copy code
bash
prefect server start
By default, Prefect stores its local data under 
~/.prefect
, including the SQLite DB, so your flow history and other metadata persist across restarts. 3. Open the UI
Copy code
text
<http://127.0.0.1:4200>
4. In another terminal, point your CLI to the local server
Copy code
bash
prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>
5. Create a local work pool For a single laptop, 
process
is usually easiest. First, verified CLI help exists for these commands would normally be ideal, but the practical pattern is:
Copy code
bash
prefect work-pool create my-local-pool --type process
6. Create a simple flow
Copy code
python
from prefect import flow

@flow(log_prints=True)
def hello():
    print("hello from my laptop")

if __name__ == "__main__":
    hello.serve(name="hello-deployment")
7. Run it
Copy code
bash
python flow.py
That creates a deployment and starts serving it locally. You can also deploy to the work pool instead of using 
.serve()
. Example:
Copy code
python
from prefect import flow

@flow(log_prints=True)
def hello():
    print("hello from my laptop")

if __name__ == "__main__":
    hello.deploy(
        name="hello-deployment",
        work_pool_name="my-local-pool",
    )
Then start a worker:
Copy code
bash
prefect worker start --pool my-local-pool
Option B: Docker version on one laptop If you want the Prefect server itself in Docker, make sure you mount persistent storage.
Copy code
bash
docker run \
  --name prefect-server \
  -p 4200:4200 \
  -v ~/.prefect:/root/.prefect \
  prefecthq/prefect:3-latest \
  prefect server start --host 0.0.0.0
Then on your host machine:
Copy code
bash
prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>
You can still run flows/workers on the laptop outside Docker, or containerize those too if you want. Recommended single-laptop pattern For pure simplicity, I’d suggest: - run 
prefect server start
- use a 
process
work pool - run one worker locally - keep everything in one Python environment That looks like: Terminal 1:
Copy code
bash
prefect server start
Terminal 2:
Copy code
bash
prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>
prefect work-pool create my-local-pool --type process
prefect worker start --pool my-local-pool
Terminal 3:
Copy code
bash
python deploy.py
Example `deploy.py`:
Copy code
python
from prefect import flow

@flow(log_prints=True)
def hello(name: str = "world"):
    print(f"hello, {name}")

if __name__ == "__main__":
    hello.deploy(
        name="hello-local",
        work_pool_name="my-local-pool",
    )
Then trigger runs from the UI or CLI. Persistence on one laptop Your metadata persists because it lives on disk, usually in:
Copy code
text
~/.prefect/prefect.db
If you want to be extra safe, back it up occasionally:
Copy code
bash
cp ~/.prefect/prefect.db ~/.prefect/prefect.db.bak
When you later move to managed That migration path is pretty straightforward conceptually: - keep the flow code - switch API target from local server to Prefect Cloud - create a cloud work pool - redeploy flows to that pool The exact deployment model may change depending on whether you use managed, hybrid, or push pools, but your flow code itself can remain mostly the same. If you want, I can give you a copy-paste quickstart with: - one example flow - one deployment file - the exact 3 terminals to open - and the commands in order.
m
@Marvin no thanks. This is quite far from what I want to do, since I still want to use Docker and Terraform. I think I'll figure it out on my own from now on, we've been going around in circles.
m
✅ thought for 11.5 seconds
Understood — and fair enough. Given your requirements, the concise conclusion is: - If you want Prefect Cloud + your own Docker image + Terraform-managed infrastructure, you need a paid tier that supports hybrid or push work pools. - If you want to stay self-hosted and still use Docker + Terraform, that’s possible, but it becomes more of an infrastructure project than a lightweight hobby setup. Sorry for the circular path getting there. For reference, the most relevant docs are: - Pricing - Work pools concepts - Managed work pools - Serverless / push infrastructure If you come back later with a specific question like: - “how do I model a self-hosted Prefect setup in Terraform?” - “what work pool type maps best to Docker on one VM?” - “how do I deploy a flow from a custom image in Prefect 3?” —I can keep it narrow and direct.
3 Views
Open in Slack
PreviousNext
Powered by