https://prefect.io logo
Title
n

Nathan Low

04/06/2023, 12:46 PM
Hello all, I'm seeing the strangest error when pulling down flows from a samba block, it's working in two other environments, and permissions to the shared folder look ok. I've trying getting debug logs and putting the api in debug mode, but I'm not seeing any error messages:
Downloading flow code from storage at 'EzeCastle'
08:11:02 AM
DEBUG
Importing flow code from 'ezecastle_daily_p_and_l_loader.py:hourly_loader'
08:11:02 AM
ERROR
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 846, in exec_module
File "<frozen importlib._bootstrap_external>", line 982, in get_code
File "<frozen importlib._bootstrap_external>", line 1039, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'ezecastle_daily_p_and_l_loader.py'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\PrefectAgent\.venv\lib\site-packages\prefect\engine.py", line 268, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "C:\PrefectAgent\.venv\lib\site-packages\prefect\client\utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "C:\PrefectAgent\.venv\lib\site-packages\prefect\deployments.py", line 187, in load_flow_from_flow_run
flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
File "C:\PrefectAgent\.venv\lib\site-packages\prefect\utilities\asyncutils.py", line 91, in run_sync_in_worker_thread
return await <http://anyio.to|anyio.to>_thread.run_sync(
File "C:\PrefectAgent\.venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\PrefectAgent\.venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "C:\PrefectAgent\.venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
result = context.run(func, *args)
File "C:\PrefectAgent\.venv\lib\site-packages\prefect\flows.py", line 786, in load_flow_from_entrypoint
flow = import_object(entrypoint)
File "C:\PrefectAgent\.venv\lib\site-packages\prefect\utilities\importtools.py", line 195, in import_object
module = load_script_as_module(script_path)
File "C:\PrefectAgent\.venv\lib\site-packages\prefect\utilities\importtools.py", line 158, in load_script_as_module
raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'ezecastle_daily_p_and_l_loader.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
n

Nelson Griffiths

04/06/2023, 1:30 PM
I actually just started running into this yesterday after updating a deployment that has been successfully running for weeks. The curious thing is that I have another deployment in the same repository running on the same environment that works. I haven't found an answer yet. Wondering if there is an easy way to reproduce what the agent does when pulling the flow from storage so I can see what exactly is being pulled and why it can't find the file?
n

Nathan Low

04/06/2023, 1:51 PM
Yeah I'd like to see what it's doing. I've looked at the code behind it:
Just knowing which directory it's trying to look at would be very helpful.
f

Filip Panovski

04/06/2023, 3:24 PM
Are you by any chance using the
path
kwarg when creating your deployments? I had some issues with that some time ago as well. Since my repo is (still) small, I ended up just placing all my flow code on the root level and omitted the
path
kwarg.
n

Nelson Griffiths

04/06/2023, 4:22 PM
I am using
path
. I actually have a need for it unless I want to set up a separate bucket for each flow, which I don't currently want to. @Nathan Low I fixed this by simply changing my
path
arg and deploying again. No clue why this worked. Seems like a bug but also have no idea how to reproduce it.
🤔 1
@Jeff Hale I saw your 🤔... I am actually starting to find that in the past 48 hours a lot of deployments that were working before have started seeing this same error.
j

Jeff Hale

04/07/2023, 3:59 PM
@Nathan Low, what’s your
prefect version
?
n

Nathan Low

04/07/2023, 4:04 PM
It's 2.7.12 I believe. Ended up trying local and putting the direct path in, that is "working" but not ideal. Even putting in a path directly to the share didn't work, even when the share is on the local computer itself. Our sysadmin team thinks group policies could be the reason why, but I'm not able to see any logs or debug notes to see where to start.
I'll get my work laptop to double check, brb
2.7.12, yep. Would an upgrade fix this?
j

Jeff Hale

04/07/2023, 4:32 PM
Perhaps. I'm not sure.
@Nelson Griffiths what is your
prefect version
result?
n

Nelson Griffiths

04/07/2023, 4:35 PM
@Jeff Hale This just got fixed in the
2.10.2
release. Was chatting in another thread about this with @Chris White and he got things moving and a bug fix in.
🙌 2
j

Jeff Hale

04/07/2023, 4:41 PM
Sweet!
n

Nathan Low

04/07/2023, 8:55 PM
I’ll try an upgrade later and see if that helps, thanks! If it doesn’t, is there a good way to debug the path it’s looking at? I guess it would be a combination of the storage block details and the path, but I can’t see it.
Tried the update, still no dice. Is there any way I can debug this more or see any more error messages @Jeff Hale?
j

Jeff Hale

04/10/2023, 3:12 PM
from the top line of your screen shot it shows the path it’s looking a @Nathan Low. Is
ice_liquidity_loader.py
directly under the
ICE
directory?
n

Nathan Low

04/10/2023, 3:14 PM
Yeah, it is. I've tried full path, tried just ICE, but no luck.
\\alphasimplex.com\Directories\Development\Prefect_Storage\prod\ICE
c

Chris White

04/10/2023, 3:16 PM
When you upgraded did you also begin using the new projects format? The fix Nelson was referencing also required that change along with the upgrade - I can write up a few commands for you in a moment that should work
n

Nathan Low

04/10/2023, 3:27 PM
Oh? I didn't try that. What do I need to do that?
Here's my flow deployment code so far:
# import ice_liquidity_loader
from prefect.deployments import Deployment
from prefect.filesystems import SMB
import ice_liquidity_loader
from prefect.orion.schemas.schedules import CronSchedule
# Schedule Info
# <https://crontab.guru/#0,30_11-12_*_*_1-5>
# "At minute 0 and 30 past every hour from 11 through 12 on every day-of-week from Monday through Friday."
storage = SMB.load("local-code")  # load a pre-defined block
schedule = CronSchedule(cron="0,30 11-12 * * 1-5", timezone="America/New_York")
__path = "ICE"
file_name = "liquidity"
__tags = [f"{__path} Loaders"]
__work_queue_name = "QUEUE"
__first_file_date = "2019-05-02"
deployment = Deployment.build_from_flow(
flow=ice_liquidity_loader.ice_liquidity_loader_date_range,
name="Load All Time",
# tags=["test"],
parameters={"start_date": __first_file_date},
version="1",
work_queue_name=__work_queue_name,
storage=storage,
skip_upload=False,
path=__path,
tags=__tags,
)
Hmmm, how well do projects play with python deploys?
At this point we're thinking its a GPO problem on our side, but would like better ways to debug what it's doing when it tries to get the flow code.
c

Chris White

04/10/2023, 3:31 PM
for a basic local deployment (which I think is what you have here, although I'm not familiar with the SMB block which could be the source of your issue), you can do something like:
prefect project init --recipe local

prefect deploy ice_liquidity_loader.py:ice_liquidity_loader_date_range -n "Load All Time" -q "QUEUE" -p "local-pool" -version "1"
The only thing to note is that this will require a worker (which is a typed version of an agent), so in this case you'd need to start a local worker:
prefect worker start -p local-pool -t process
n

Nathan Low

04/10/2023, 3:32 PM
So there's no way to get any more debug info than this?
c

Chris White

04/10/2023, 3:50 PM
Not sure with the SMB block, I'm not familiar with it; I can tell you what's happening under the hood though: • flow code is downloaded using your block (it appears that succeeds) • next the flow function is imported (this is where you're failing) So to me it looks like the SMB block is not functioning properly - it could be that it isn't downloading anything at all, or it is downloading stuff but putting it in the wrong place
n

Nathan Low

04/10/2023, 3:56 PM
Thanks, would you know where it tries to download to? I figured it would be set by the "PREFECT_LOCAL_STORAGE_PATH" variable but that doesn't seem to be the case.
c

Chris White

04/10/2023, 4:00 PM
if you're using a process infra block then it uses a temporary directory - oftentimes if you turn on DEBUG level logging on the agent you can see more about the temporary location it's creating
n

Nelson Griffiths

04/10/2023, 4:37 PM
@Chris White I actually did not make any updates to use the new project format. I just upgraded and my previous patterns began working again.
🤯 1
c

Chris White

04/10/2023, 4:43 PM
very interesting, good to know!
n

Nathan Low

04/10/2023, 4:54 PM
We're having no luck finding the temp files, we did find a C:\Users\svc_prefect_p\AppData\Local\Temp folder, but nothing prefect related there. It's a windows server 2022 box, if that helps.
c

Chris White

04/10/2023, 5:18 PM
Yea temp files will get cleaned up immediately after completion (which in your case would be right after it errors)
could you try using the local project recipe and seeing if that works?
n

Nathan Low

04/10/2023, 5:22 PM
Yeah it does, when I point the path directly at the folder the flow py file is in. Not ideal when we have multiple agents, but it's something.
c

Chris White

04/10/2023, 5:29 PM
what issue arises when you have multiple agents?
n

Nathan Low

04/10/2023, 5:35 PM
Just need to install the flows on multiple boxes
c

Chris White

04/10/2023, 5:52 PM
ah gotcha, yea in that case basing your project out of cloud storage / github / docker would make more sense
n

Nathan Low

04/10/2023, 6:35 PM
Hmmm, after the upgrade local doesn't appear to be working anymore: Before Upgrade and after upgrade images attached.
Ran an agent until my own username and the shared samba flow locations seems to be fine. I'm thinking this is a permissions needed by the agent user thing now.
c

Chris White

04/10/2023, 10:03 PM
this is a little subtle, but are you using an agent or a worker? Assuming you are running these all on the same machine, I'm suprised to see that last error
n

Nathan Low

04/10/2023, 10:05 PM
Agent as nssm service. We ended up making the user running the service a local admin and that got it to work. Is there a page that shows permissions a user running an agent service needs on a windows box?
Rolled back to 2.7.12 so that ssl would work as well, seems like it didn't play nice with our ssl reverse proxy on iis.
c

Chris White

04/10/2023, 10:10 PM
interesting - no we don't have that documented but it's a great request; would you mind opening an issue for it?
n

Nathan Low

04/10/2023, 10:12 PM
Yeah for sure, just on GitHub yeah?
:thank-you: 1
c

Chris White

04/10/2023, 10:19 PM
yup!
n

Nathan Low

05/17/2023, 3:24 PM
Sorry it took a while, but here's the ticket: https://github.com/PrefectHQ/prefect/issues/9601