Morning all, I have a deployment scheduled which seems to be working majority of the time, but rando...

Ash

04/27/2023, 8:26 AM

Morning all, I have a deployment scheduled which seems to be working majority of the time, but randomly sometimes it'll fail with the below error. I'm struggling to understand why the file/directory wouldn't exist when the same flow is working fine at other times. Any suggestions on where to start?

Flow could not be retrieved from deployment.

Traceback (most recent call last):

File "/usr/local/lib/python3.8/dist-packages/prefect/engine.py", line 276, in retrieve_flow_then_begin_flow_run

flow = await load_flow_from_flow_run(flow_run, client=client)

File "/usr/local/lib/python3.8/dist-packages/prefect/client/utilities.py", line 40, in with_injected_client

return await fn(*args, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/prefect/deployments.py", line 195, in load_flow_from_flow_run

await storage_block.get_directory(from_path=deployment.path, local_path=".")

File "/usr/local/lib/python3.8/dist-packages/prefect/filesystems.py", line 147, in get_directory

copytree(from_path, local_path, dirs_exist_ok=True)

File "/usr/lib/python3.8/shutil.py", line 557, in copytree

return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,

File "/usr/lib/python3.8/shutil.py", line 513, in _copytree

raise Error(errors)

shutil.Error: [('/home/user/.prefect/prefect.db-shm', '/tmp/tmp791im88hprefect/.prefect/prefect.db-shm', "[Errno 2] No such file or directory: '/home/user/.prefect/prefect.db-shm'")]

flapili

04/27/2023, 8:29 AM

I don't have any hint about this error, but I would recommand to switch to postgres if it's for production

flapili

04/27/2023, 8:29 AM

even for local dev booting a psql container isn't bad

Ash

04/27/2023, 8:30 AM

Just development environment atm, but definitely something I'll look into. Thanks

flapili

04/27/2023, 8:31 AM

sound like the temp folder is deleted before getting copied as prefect.db-shm

flapili

04/27/2023, 8:32 AM

something else is deleting this folder

flapili

04/27/2023, 8:32 AM

could be the OS, a cleaning app, a cleaning routine, ect

Ash

04/27/2023, 8:32 AM

Does Prefect delete the tmp folders after the flow run has completed?

Ash

04/27/2023, 8:34 AM

I noticed the tmp folders were including the whole folder including a lot of vscode-plugins and thought that strange, as it takes up a lot of memory. I changed the upcoming schedules to be a maximum of 10 to stop it creating many tmp folders

flapili

04/27/2023, 8:34 AM

idk, according to the paths you are on linux right ?

flapili

04/27/2023, 8:34 AM

not a container ?

Ash

04/27/2023, 8:34 AM

correct

flapili

04/27/2023, 8:34 AM

you could look at auditctl tools

flapili

04/27/2023, 8:35 AM

it permit to track what change files/folders

flapili

04/27/2023, 8:35 AM

it will tell you as example what folder was deleted by what process in /tmp

flapili

04/27/2023, 8:36 AM

but it's probably overthinking 😅

flapili

04/27/2023, 8:36 AM

but anyway if you are curious you could take a look at

Ash

04/27/2023, 8:36 AM

Great, will take a look 🙂 I'm thinking maybe I can change the local storage to be something other than tmp, if it is indeed something clearing out tmp folders before prefect can access

flapili

04/27/2023, 8:37 AM

not sure, prefect probably use tempfile stdlib

Ash

04/27/2023, 8:38 AM

I will attempt postgres first as that seems to be the best practice!

Ash

04/27/2023, 8:38 AM

Thanks again for the help

9 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.