Morning all, I have a deployment scheduled which s...
# ask-communitya
Ash
04/27/2023, 8:26 AMMorning all, I have a deployment scheduled which seems to be working majority of the time, but randomly sometimes it'll fail with the below error. I'm struggling to understand why the file/directory wouldn't exist when the same flow is working fine at other times. Any suggestions on where to start?
Flow could not be retrieved from deployment.Traceback (most recent call last):File "/usr/local/lib/python3.8/dist-packages/prefect/engine.py", line 276, in retrieve_flow_then_begin_flow_runflow = await load_flow_from_flow_run(flow_run, client=client)File "/usr/local/lib/python3.8/dist-packages/prefect/client/utilities.py", line 40, in with_injected_clientreturn await fn(*args, **kwargs)File "/usr/local/lib/python3.8/dist-packages/prefect/deployments.py", line 195, in load_flow_from_flow_runawait storage_block.get_directory(from_path=deployment.path, local_path=".")File "/usr/local/lib/python3.8/dist-packages/prefect/filesystems.py", line 147, in get_directorycopytree(from_path, local_path, dirs_exist_ok=True)File "/usr/lib/python3.8/shutil.py", line 557, in copytreereturn _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,File "/usr/lib/python3.8/shutil.py", line 513, in _copytreeraise Error(errors)shutil.Error: [('/home/user/.prefect/prefect.db-shm', '/tmp/tmp791im88hprefect/.prefect/prefect.db-shm', "[Errno 2] No such file or directory: '/home/user/.prefect/prefect.db-shm'")]f
flapili
04/27/2023, 8:29 AMI don't have any hint about this error, but I would recommand to switch to postgres if it's for production
flapili
04/27/2023, 8:29 AMeven for local dev booting a psql container isn't bad
a
Ash
04/27/2023, 8:30 AMJust development environment atm, but definitely something I'll look into. Thanks
f
flapili
04/27/2023, 8:31 AMsound like the temp folder is deleted before getting copied as prefect.db-shm
flapili
04/27/2023, 8:32 AMsomething else is deleting this folder
flapili
04/27/2023, 8:32 AMcould be the OS, a cleaning app, a cleaning routine, ect
a
Ash
04/27/2023, 8:32 AMDoes Prefect delete the tmp folders after the flow run has completed?
Ash
04/27/2023, 8:34 AMI noticed the tmp folders were including the whole folder including a lot of vscode-plugins and thought that strange, as it takes up a lot of memory. I changed the upcoming schedules to be a maximum of 10 to stop it creating many tmp folders
f
flapili
04/27/2023, 8:34 AMidk, according to the paths you are on linux right ?
flapili
04/27/2023, 8:34 AMnot a container ?
a
Ash
04/27/2023, 8:34 AMcorrect
f
flapili
04/27/2023, 8:34 AMyou could look at auditctl tools
flapili
04/27/2023, 8:35 AMit permit to track what change files/folders
flapili
04/27/2023, 8:35 AMit will tell you as example what folder was deleted by what process in /tmp
flapili
04/27/2023, 8:36 AMbut it's probably overthinking 😅
flapili
04/27/2023, 8:36 AMbut anyway if you are curious you could take a look at
a
Ash
04/27/2023, 8:36 AMGreat, will take a look 🙂 I'm thinking maybe I can change the local storage to be something other than tmp, if it is indeed something clearing out tmp folders before prefect can access
f
flapili
04/27/2023, 8:37 AMnot sure, prefect probably use tempfile stdlib
a
Ash
04/27/2023, 8:38 AMI will attempt postgres first as that seems to be the best practice!
Ash
04/27/2023, 8:38 AMThanks again for the help
9 Views