Bob Peers
12/30/2025, 1:57 PMMarvin
12/30/2025, 1:57 PMMarvin
12/30/2025, 2:02 PMflow.from_source(<local path>), Prefect adds an implicit pull step that just calls os.chdir(<that path>) on the worker before importing your flow. If that directory does not exist at the exact moment the run starts on the worker, you’ll see the FileNotFoundError from prefect.deployments.steps.set_working_directory.
Why it can be intermittent:
- The deployment stores the path as a string; the worker must have that exact directory present at runtime. If you have more than one worker (or workers on different machines), some workers may not have that path.
- If the worker is running under a different user/session (e.g., Windows Service vs. your interactive user), it may not see the same filesystem. On Windows, services also can’t see user-mapped drives (use UNC paths if it’s a share).
- The directory may be moved/renamed/cleaned up between deployment creation and execution (even briefly) — os.chdir will fail immediately if the path isn’t there.
There’s a related report (2.x) that shows this implicit step in action: Encountered error while running prefect.deployments.steps.set_working_directory - FileNotFoundError
Suggestions to make this robust:
- Verify scope and workers
- If you have multiple workers in the pool, ensure the directory exists on every worker host, at the same path.
- Confirm the worker is running under the same user context that has access to that folder (especially if the worker is a Windows Service).
- If this is a network/share location, prefer a UNC path like \\server\share\workflows_prefectOSS over a mapped drive letter.
- Keep the path stable and absolute
- Use an absolute path (you are), and ensure it’s persistent (not in a temp or ephemeral location).
- Make sure nothing is moving/renaming that directory while runs start.
- Consider switching storage to avoid local path dependency
- Use from_source("<https://github.com/owner/repo>") so the worker clones the repo at runtime (most robust across workers).
- Or install your flows as a package on the worker’s environment and deploy without a local-path source (so there’s no working-directory pull step needed).
If you want to double-check what’s stored in your deployment, you can inspect the deployment’s pull_steps — you’ll see the set_working_directory with your path.
A couple quick questions to help pinpoint the cause:
- Which Prefect version are you on (2.x or 3.x, and exact version)?
- Is there more than one worker in this work pool? Are they all on the same machine?
- Is C:\workflows_prefectOSS local SSD/HDD, or is it a folder that’s synced/mirrored by another tool?
- How are you starting the worker (interactive shell vs. Windows Service)?
- Can you share how you’re creating the deployment (your from_source(..., entrypoint=...) call)?
If you’d like, paste your deployment code snippet and worker setup details and I’ll help you tighten it up.