https://prefect.io logo
Title
s

Stéphan Taljaard

07/04/2021, 4:22 PM
Hi. Any pointers for limiting the server postgresql db size? Any specific things to watch out for while truncating the tables?
Turns out the big culprit was actually not the database...
prefect_server=# \l+
                                                                    List of databases
      Name      |  Owner  | Encoding |  Collate   |   Ctype    |  Access privileges  |  Size   | Tablespace |                Description
----------------+---------+----------+------------+------------+---------------------+---------+------------+--------------------------------------------
 postgres       | prefect | UTF8     | en_US.utf8 | en_US.utf8 |                     | 7521 kB | pg_default | default administrative connection database
 prefect_server | prefect | UTF8     | en_US.utf8 | en_US.utf8 |                     | 1083 MB | pg_default |
 template0      | prefect | UTF8     | en_US.utf8 | en_US.utf8 | =c/prefect         +| 7521 kB | pg_default | unmodifiable empty database
                |         |          |            |            | prefect=CTc/prefect |         |            |
 template1      | prefect | UTF8     | en_US.utf8 | en_US.utf8 | =c/prefect         +| 7521 kB | pg_default | default template for new databases
                |         |          |            |            | prefect=CTc/prefect |         |            |
(4 rows)
prefect@data-integration:~$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
udev             1012520       0   1012520   0% /dev
tmpfs             204360   21236    183124  11% /run
/dev/sda1       10126600 9379576    212908  98% /
tmpfs            1021796      16   1021780   1% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs            1021796       0   1021796   0% /sys/fs/cgroup
/dev/sda15        126710    6148    120562   5% /boot/efi
overlay         10126600 9379576    212908  98% /var/lib/docker/overlay2/76286bab1d42d570a9adc9b810d88bc21e63b4d325447af9bdb75a930d08a20e/merged
overlay         10126600 9379576    212908  98% /var/lib/docker/overlay2/6c70c2215f12f1fed0c2ff9a7eabff0a1834816e135c56cd63b1c1a6fe9becb3/merged
overlay         10126600 9379576    212908  98% /var/lib/docker/overlay2/0ca7b9d98a8d87f2ab76a0e34211f93110b26ce23efcd533517fb8b2d6b0f362/merged
overlay         10126600 9379576    212908  98% /var/lib/docker/overlay2/e64581578f1140a39de54d4dc9e3999b9d928804a06a5538210867dce87d7e8b/merged
overlay         10126600 9379576    212908  98% /var/lib/docker/overlay2/7a7aa1e9979ddd4ba92afae315895bb3a7bf4000fcc9dbfcabc08c8fe064a4f5/merged
overlay         10126600 9379576    212908  98% /var/lib/docker/overlay2/51be48a363e4f01bee5ca8b0bfb1868caa06ef0eb50f7f5b9023d817e909b7c4/merged
tmpfs             204356       0    204356   0% /run/user/1001
I haven't really figured this out yet. Will update here with my findings.
Okay I'm finding multiple sources of files piling up. Is it safe to delete old files in
.prefect/results
manually? (I'm using LocalAgent with defaults, and no result caching and not making use of checkpointing)
k

Kevin Kho

07/06/2021, 10:51 AM
Hey @Stéphan Taljaard, yes you can. Some people use the same result location so that newer copies of the flow overwrite the same file and only the latest result is stored
s

Stéphan Taljaard

07/06/2021, 11:37 AM
Thanks! That sounds like a good idea