<@ULVA73B9P> I am looking to implement a CI/CD pip...
# ask-marvin
e
@Marvin I am looking to implement a CI/CD pipeline for my prefect server/workers running on my remote windows server. My plan is: When a developer pushes to main branch, a github actions workflow pulls the latest changes on the remote server. This installs any new dependencies etc. My big concern is what happens if a developer pushes code to the main branch WHILE existing flows are running. Will any errors be caused?
m
thought for 393.5 seconds
Great question. Short answer: if you update code in-place on the same machine/venv that’s running your flows, you can cause inconsistent imports and runtime errors in in‑flight runs. There are safe patterns to avoid this. Assumptions: Prefect 3.x. If you’re on 2.x, let me know and I’ll adjust. What happens if you git pull + pip install during a run? - If your deployments execute code from a shared directory and shared virtualenv that you update in-place, there’s risk: - Python processes don’t reload already-imported modules, but any imports that happen later in the run may load the new/changed code or dependencies. - pip installing over a running interpreter can temporarily remove/replace packages, causing ImportError/AttributeError mid-run. - On Windows, file locking can also make updates fail partially. - If your deployments fetch an immutable source checkout per run (recommended), the running flow uses its own working directory and is isolated from subsequent updates. Safe patterns to prevent breakage 1) Pin deployments to an immutable git commit - Make deployments fetch an exact commit for every run so new pushes do not affect in-flight runs. - Use pull steps in prefect.yaml to pin a commit SHA:
Copy code
# prefect.yaml (excerpt)
  deployments:
    - name: prod
      entrypoint: flows/my_flow.py:my_flow
      pull:
        - prefect.deployments.steps.pull.git_clone:
            repository: "<https://github.com/your-org/your-repo.git>"
            commit_sha: "abc123def456..."   # pin to the exact commit
            id: "clone"
        - prefect.deployments.steps.pull.set_working_directory:
            directory: "{{ clone.directory }}"
- Then deploy from the repo state you want to freeze:
Copy code
prefect deploy
- Each new release = new commit + new deployment (or new deployment version). Existing runs continue on the pinned code; new runs use the new deployment. Docs: - prefect.yaml reference - Deployment versioning - Pull steps API: prefect.deployments.steps.pull 2) Use containers (most robust) - Build an image per release with your code + dependencies baked in; point your deployment at that image. Every run uses a frozen image. - Works great with Docker or Kubernetes work pools. - If you’re on Windows Server, verify whether you can run Linux containers (common) or Windows containers in your environment. 3) Blue/green directories + pause/resume - Keep two directories on the server (e.g., C:\apps\myflow-A and C:\apps\myflow-B). Workers run from A while you prepare B. Switch workers to B when ready. - For a safe switchover window, pause new work before updating: - Pause the work pool:
Copy code
prefect work-pool pause <pool-name>
- Or pause deployment schedules (stops auto-scheduling; manual runs still possible):
Copy code
prefect deployment schedule pause <flow-name>/<deployment-name>
- Wait for in-flight runs to finish, update code/deps, then:
Copy code
prefect work-pool resume <pool-name>
- You can also set a work pool concurrency limit to 0 temporarily:
Copy code
prefect work-pool set-concurrency-limit <pool-name> 0
Docs: - Work pools 4) Separate virtualenv per release - Create a new venv for each release and point your worker or deployment job variables at that venv. Never pip install into the same venv used by running flows. Windows-specific notes - Real-time antivirus can interfere with file operations; consider excluding your Python install, %USERPROFILE%\.prefect, and your working directories. - File locks can make in-place updates brittle—another reason to prefer pinned checkouts, separate venvs, or containers.
- Self-hosted server on Windows: Windows guide Putting it together: a safe CI/CD flow (no containers) - On push to main, your CI: - Builds/tests code - Creates/updates a deployment pinned to that commit (via
prefect.yaml
pull steps with
commit_sha
) - Optionally pauses your work pool, waits for a drain window, and resumes - Workers keep running; each new flow run pulls the pinned code for its deployment into its own working dir. In-flight runs are unaffected. If you share how you’re deploying today: - Are you using
prefect deploy
from a git repo, or running from a shared directory on the Windows box? - Are you using a single shared virtualenv or per-project venvs? - Any ability to run Docker/Kubernetes? I can then sketch a concrete GitHub Actions workflow and Prefect config tailored to your setup.