Hello all, I'm seeing the strangest error when pulling down flows from a samba block, it's working in two other environments, and permissions to the shared folder look ok. I've trying getting debug logs and putting the api in debug mode, but I'm not seeing any error messages:

Downloading flow code from storage at 'EzeCastle'

08:11:02 AM

DEBUG

Importing flow code from 'ezecastle_daily_p_and_l_loader.py:hourly_loader'

08:11:02 AM

ERROR

Flow could not be retrieved from deployment.

Traceback (most recent call last):

File "<frozen importlib._bootstrap_external>", line 846, in exec_module

File "<frozen importlib._bootstrap_external>", line 982, in get_code

File "<frozen importlib._bootstrap_external>", line 1039, in get_data

FileNotFoundError: [Errno 2] No such file or directory: 'ezecastle_daily_p_and_l_loader.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "C:\PrefectAgent\.venv\lib\site-packages\prefect\engine.py", line 268, in retrieve_flow_then_begin_flow_run

flow = await load_flow_from_flow_run(flow_run, client=client)

File "C:\PrefectAgent\.venv\lib\site-packages\prefect\client\utilities.py", line 47, in with_injected_client

return await fn(*args, **kwargs)

File "C:\PrefectAgent\.venv\lib\site-packages\prefect\deployments.py", line 187, in load_flow_from_flow_run

flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))

File "C:\PrefectAgent\.venv\lib\site-packages\prefect\utilities\asyncutils.py", line 91, in run_sync_in_worker_thread

return await <http://anyio.to|anyio.to>_thread.run_sync(

File "C:\PrefectAgent\.venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync

return await get_asynclib().run_sync_in_worker_thread(

File "C:\PrefectAgent\.venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread

return await future

File "C:\PrefectAgent\.venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run

result = context.run(func, *args)

File "C:\PrefectAgent\.venv\lib\site-packages\prefect\flows.py", line 786, in load_flow_from_entrypoint

flow = import_object(entrypoint)

File "C:\PrefectAgent\.venv\lib\site-packages\prefect\utilities\importtools.py", line 195, in import_object

module = load_script_as_module(script_path)

File "C:\PrefectAgent\.venv\lib\site-packages\prefect\utilities\importtools.py", line 158, in load_script_as_module

raise ScriptError(user_exc=exc, path=path) from exc

prefect.exceptions.ScriptError: Script at 'ezecastle_daily_p_and_l_loader.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')

I actually just started running into this yesterday after updating a deployment that has been successfully running for weeks. The curious thing is that I have another deployment in the same repository running on the same environment that works. I haven't found an answer yet. Wondering if there is an easy way to reproduce what the agent does when pulling the flow from storage so I can see what exactly is being pulled and why it can't find the file?

Yeah I'd like to see what it's doing. I've looked at the code behind it:

Are you by any chance using the

path

kwarg when creating your deployments? I had some issues with that some time ago as well. Since my repo is (still) small, I ended up just placing all my flow code on the root level and omitted the

path

kwarg.

I am using

path

. I actually have a need for it unless I want to set up a separate bucket for each flow, which I don't currently want to. @Nathan Low I fixed this by simply changing my

path

arg and deploying again. No clue why this worked. Seems like a bug but also have no idea how to reproduce it.

@Nathan Low, what’s your

prefect version

?

It's 2.7.12 I believe. Ended up trying local and putting the direct path in, that is "working" but not ideal. Even putting in a path directly to the share didn't work, even when the share is on the local computer itself. Our sysadmin team thinks group policies could be the reason why, but I'm not able to see any logs or debug notes to see where to start.

Perhaps. I'm not sure.

@Jeff Hale This just got fixed in the

2.10.2

release. Was chatting in another thread about this with @Chris White and he got things moving and a bug fix in.

Sweet!

I’ll try an upgrade later and see if that helps, thanks! If it doesn’t, is there a good way to debug the path it’s looking at? I guess it would be a combination of the storage block details and the path, but I can’t see it.

from the top line of your screen shot it shows the path it’s looking a @Nathan Low. Is

ice_liquidity_loader.py

directly under the

ICE

directory?

Yeah, it is. I've tried full path, tried just ICE, but no luck.

When you upgraded did you also begin using the new projects format? The fix Nelson was referencing also required that change along with the upgrade - I can write up a few commands for you in a moment that should work

Oh? I didn't try that. What do I need to do that?

for a basic local deployment (which I think is what you have here, although I'm not familiar with the SMB block which could be the source of your issue), you can do something like:

prefect project init --recipe local

prefect deploy ice_liquidity_loader.py:ice_liquidity_loader_date_range -n "Load All Time" -q "QUEUE" -p "local-pool" -version "1"

The only thing to note is that this will require a worker (which is a typed version of an agent), so in this case you'd need to start a local worker:

prefect worker start -p local-pool -t process

So there's no way to get any more debug info than this?

Not sure with the SMB block, I'm not familiar with it; I can tell you what's happening under the hood though: • flow code is downloaded using your block (it appears that succeeds) • next the flow function is imported (this is where you're failing) So to me it looks like the SMB block is not functioning properly - it could be that it isn't downloading anything at all, or it is downloading stuff but putting it in the wrong place

Thanks, would you know where it tries to download to? I figured it would be set by the "PREFECT_LOCAL_STORAGE_PATH" variable but that doesn't seem to be the case.

if you're using a process infra block then it uses a temporary directory - oftentimes if you turn on DEBUG level logging on the agent you can see more about the temporary location it's creating

@Chris White I actually did not make any updates to use the new project format. I just upgraded and my previous patterns began working again.

very interesting, good to know!

We're having no luck finding the temp files, we did find a C:\Users\svc_prefect_p\AppData\Local\Temp folder, but nothing prefect related there. It's a windows server 2022 box, if that helps.

Yea temp files will get cleaned up immediately after completion (which in your case would be right after it errors)

Yeah it does, when I point the path directly at the folder the flow py file is in. Not ideal when we have multiple agents, but it's something.

what issue arises when you have multiple agents?

Just need to install the flows on multiple boxes

ah gotcha, yea in that case basing your project out of cloud storage / github / docker would make more sense

Hmmm, after the upgrade local doesn't appear to be working anymore: Before Upgrade and after upgrade images attached.

this is a little subtle, but are you using an agent or a worker? Assuming you are running these all on the same machine, I'm suprised to see that last error

Agent as nssm service. We ended up making the user running the service a local admin and that got it to work. Is there a page that shows permissions a user running an agent service needs on a windows box?

interesting - no we don't have that documented but it's a great request; would you mind opening an issue for it?

Yeah for sure, just on GitHub yeah?

yup!

Sorry it took a while, but here's the ticket: https://github.com/PrefectHQ/prefect/issues/9601