<@ULVA73B9P> how to bundle python + prefect into a...
# ask-marvin
p
@Marvin how to bundle python + prefect into a single EXE that is easily distributable for Windows servers?
m
thought for 319.3 seconds
Short answer: Prefect doesn’t officially support or recommend bundling into a single Windows EXE. In Prefect 3.x we encourage shipping your code in a container or installing Python on the server and running a Process worker. That said, if a single EXE is a hard requirement, you can usually get a minimal worker to run with PyInstaller—just expect some trial-and-error due to Prefect’s dynamic imports. Recommended approaches (from most reliable to most brittle) 1) Preferred: Containerize - Ship a Docker image that contains Python, Prefect, and your flow code. Run a Process worker in the container on the Windows host or in an orchestrator. - Docs: Docker deployment and Serve flows in Docker 2) Also reliable: Install Python + venv on the server - Install Python (or use the Windows embeddable distribution), create a virtualenv,
pip install prefect
and your dependencies, then start a Process worker or serve your flows. - Workers concept: Workers overview - Windows server notes (if you’re self-hosting): Windows server guide 3) If you must produce a single EXE (unsupported but doable) Key idea: avoid dynamic entrypoints and the full CLI; embed a small, static wrapper that starts a Process worker and let Prefect deliver flow runs from your work pool. A. Minimal wrapper script (start a Process worker)
Copy code
# run_worker.py
import os
from prefect.workers.process import ProcessWorker

def main():
    pool = os.environ.get("PREFECT_WORK_POOL")
    if not pool:
        raise SystemExit("Set PREFECT_WORK_POOL to the name of your work pool.")
    queues = os.environ.get("PREFECT_WORK_QUEUES")
    work_queues = [q.strip() for q in queues.split(",")] if queues else None

    worker = ProcessWorker(
        work_pool_name=pool,
        work_queues=work_queues,
        name=os.environ.get("PREFECT_WORKER_NAME"),
    )
    # Optional: healthcheck endpoint for service monitoring
    worker.start(with_healthcheck=True)

if __name__ == "__main__":
    main()
B. Build with PyInstaller - Pin exact versions of Python/Prefect/dependencies for repeatable builds. - Collect Prefect and heavy deps so dynamic/lazy imports are included; include certifi data for TLS. - Example (PowerShell, line breaks for readability):
Copy code
pyinstaller --onefile --name prefect-worker `
  --collect-all prefect `
  --collect-all anyio `
  --collect-all httpx `
  --collect-all sqlalchemy `
  --collect-all pydantic `
  --collect-all rich `
  --collect-all click `
  --collect-data certifi `
  --hidden-import=sniffio `
  --hidden-import=cloudpickle `
  run_worker.py
Notes and caveats: - Dynamic imports: Prefect loads modules and “collections” dynamically. The
--collect-all prefect
flag helps, but you may still need to add more
--collect-all
or
--hidden-import
entries based on what your flows import. - Certificates: Without
certifi
data, HTTPS calls to Prefect Cloud/Server can fail. - Don’t use entrypoint-based flow loading in a frozen app (e.g., strings like "package.module:flow"). Import your flows directly if you’re bundling your flow code into the EXE, or better yet, just run a worker and keep your flow code accessible via your normal deployment mechanism. - CLI embedding: Trying to ship the full
prefect
CLI inside the EXE is fragile; prefer the programmatic worker wrapper above. - Test on a clean Windows VM that mimics the server. If the EXE crashes, rebuild with
--debug
to reveal missing modules. - Running as a service: Consider using NSSM or Windows Services to keep the EXE running and auto-restart. When to consider alternatives like zipapps/PEX/PyOxidizer/Nuitka - Zipapps (shiv/pex) produce a single file but still require a system Python.
- Nuitka or PyOxidizer can sometimes produce tighter single binaries, but you’ll face the same dynamic import and packaging wrinkles. These paths are more complex to get right with Prefect. Links to help you choose the path - Workers concept: Workers overview - Create deployments: Create deployments - Docker-based infra: Docker deployment - Serve in Docker: Serve flows in Docker - Windows server notes: Windows server guide A couple of questions to tailor this for you: - Is “single file EXE” a hard requirement, or would a “single folder + .bat” (portable Python + venv) be acceptable? - Are these servers offline/air-gapped, and is Docker allowed or not? - Do you want to bundle just a worker that polls a Prefect work pool, or a fully standalone app that executes specific flows on a schedule? If you can share those constraints and your Prefect version, I can provide a concrete, tested build recipe for your scenario.