Newskooler
12/10/2020, 3:28 PMprefect/flows
so that we can keep it at a manageable size (it tends to just grow naturally; and it does so quite quickly)? đ¤Zanie
12/10/2020, 3:50 PMNewskooler
12/10/2020, 3:50 PM--volume-path
 but instead itâs the task flow.storage = Local(directory='/this/location)
â Also it has grown to 100GB for less than 30 days.Dylan
12/10/2020, 4:39 PMNewskooler
12/10/2020, 4:41 PMZanie
12/10/2020, 4:42 PMflow.storage
is where the pickled flow is stored (when using Local
) â the code of your flowNewskooler
12/10/2020, 4:44 PMZanie
12/10/2020, 4:44 PMLocal
storage, the flow is simply pickled with cloudpickle
and dumped as bytes â do you have a ton of files there? Whatâs the smallest/largest file?Newskooler
12/10/2020, 4:45 PMZanie
12/10/2020, 4:46 PMNewskooler
12/10/2020, 4:47 PMZanie
12/10/2020, 4:49 PMNewskooler
12/10/2020, 5:25 PM-rw-r--r-- 1 root root 1436873 Dec 10 17:24 prefect-result-2020-12-10t17-24-10-287987-00-00
-rw-r--r-- 1 root root 1179386 Dec 10 17:24 prefect-result-2020-12-10t17-24-10-629558-00-00
-rw-r--r-- 1 root root 13969 Dec 10 17:24 prefect-result-2020-12-10t17-24-15-435020-00-00
-rw-r--r-- 1 root root 16465 Dec 10 17:24 prefect-result-2020-12-10t17-24-15-821351-00-00
-rw-r--r-- 1 root root 94 Dec 10 17:24 prefect-result-2020-12-10t17-24-16-142481-00-00
67836 ./prefect-result-2020-12-08t18-57-57-583026-00-00
and this is the smallest:
4 ./prefect-result-2020-11-30t00-00-15-595208-00-00
Zanie
12/10/2020, 5:28 PMLocal
storage class, the associated result type is defined as result = LocalResult(self.directory, validate_dir=validate)
â this means that this directory will contain both your serialized flow and all of your flow run results.Newskooler
12/10/2020, 5:29 PMZanie
12/10/2020, 5:30 PMprefect-result-*
can be safely deleted if you do not need that data anymore. Furthermore, you can disable checkpointing for tasks if you donât need the ability to resume.Newskooler
12/10/2020, 5:32 PMprefect-result-*
) older than X days, does this make sense?Zanie
12/10/2020, 5:47 PMNewskooler
12/10/2020, 5:48 PMZanie
12/10/2020, 5:49 PMNewskooler
12/10/2020, 5:50 PM