<@ULVA73B9P> I'm using a process worker on local f...
# ask-marvin
b
@Marvin I'm using a process worker on local files using from_source with a local path. It randomly works and doesn't work. When it fails I see these lines. The worked has full access to the location and sometimes works so it can't be a permissions issue: File "/usr/local/lib/python3.12/site-packages/prefect/deployments/steps/pull.py", line 35, in set_working_directory os.chdir(directory) FileNotFoundError: [Errno 2] No such file or directory: 'C:\\workflows_prefectOSS'
m
thought for 303.1 seconds
Thanks for the stack trace — that’s a helpful clue. What’s happening here: when you use
flow.from_source(<local path>)
, Prefect adds an implicit pull step that just calls
os.chdir(<that path>)
on the worker before importing your flow. If that directory does not exist at the exact moment the run starts on the worker, you’ll see the
FileNotFoundError
from
prefect.deployments.steps.set_working_directory
. Why it can be intermittent: - The deployment stores the path as a string; the worker must have that exact directory present at runtime. If you have more than one worker (or workers on different machines), some workers may not have that path. - If the worker is running under a different user/session (e.g., Windows Service vs. your interactive user), it may not see the same filesystem. On Windows, services also can’t see user-mapped drives (use UNC paths if it’s a share). - The directory may be moved/renamed/cleaned up between deployment creation and execution (even briefly) —
os.chdir
will fail immediately if the path isn’t there. There’s a related report (2.x) that shows this implicit step in action: Encountered error while running prefect.deployments.steps.set_working_directory - FileNotFoundError Suggestions to make this robust: - Verify scope and workers - If you have multiple workers in the pool, ensure the directory exists on every worker host, at the same path. - Confirm the worker is running under the same user context that has access to that folder (especially if the worker is a Windows Service). - If this is a network/share location, prefer a UNC path like
\\server\share\workflows_prefectOSS
over a mapped drive letter. - Keep the path stable and absolute - Use an absolute path (you are), and ensure it’s persistent (not in a temp or ephemeral location). - Make sure nothing is moving/renaming that directory while runs start. - Consider switching storage to avoid local path dependency - Use
from_source("<https://github.com/owner/repo>")
so the worker clones the repo at runtime (most robust across workers). - Or install your flows as a package on the worker’s environment and deploy without a local-path source (so there’s no working-directory pull step needed). If you want to double-check what’s stored in your deployment, you can inspect the deployment’s
pull_steps
— you’ll see the
set_working_directory
with your path. A couple quick questions to help pinpoint the cause: - Which Prefect version are you on (2.x or 3.x, and exact version)? - Is there more than one worker in this work pool? Are they all on the same machine? - Is
C:\workflows_prefectOSS
local SSD/HDD, or is it a folder that’s synced/mirrored by another tool? - How are you starting the worker (interactive shell vs. Windows Service)? - Can you share how you’re creating the deployment (your
from_source(..., entrypoint=...)
call)? If you’d like, paste your deployment code snippet and worker setup details and I’ll help you tighten it up.