https://prefect.io logo
Title
s

Szymon Szyszko

11/02/2022, 4:55 PM
Hi everyone 👋 My goal is to put Sqlite orion.db in EFS mounted into EC2 instance. It generates an error. Error logs are attached in the thread to keep the channel clean. According to Amazon Elastic File System (Amazon EFS) Now Supports NFSv4 Lock Upgrading and Downgrading it is possible to use Sqlite in EFS. I noticed that in src/prefect/orion/database/configurations.py#L262 you set journal_mode to
WAL
which is not allowed in network file systems. I changed that to
delete
without success - logs do not change. I have seen similar logs to mine here In my project, I have other Sqlite databases in the EFS directory and they work fine.
1
Attached logs are from k8s pod. Logs from EC2 are the same.
a

Anna Geller

11/03/2022, 12:07 AM
s

Szymon Szyszko

11/03/2022, 8:21 AM
No. I would rather find source of the problem than workaround. I did not use Docker to reproduce the problem. I run Python code on EC2 with EFS mounted to /home/ubuntu/.prefect Definitely using EFS causes the problem. When EFS is not mounted it works.
a

Anna Geller

11/03/2022, 11:55 AM
Using SQLite backend is meant more for a lightweight setup (no mounts), and if you need something more robust or bigger scale, you would either use PostgreSQL or point your profile to Prefect Cloud.
s

Szymon Szyszko

11/03/2022, 12:29 PM
I do not need a bigger scale. SQLite fits my load needs. I don't know how placing SQLite in network mount makes a solution not lightweight. It can but in my case, it will not.
a

Anna Geller

11/03/2022, 12:33 PM
can you tell me why you want to put SQLite on EFS, then? you can still mount an EFS volume to your EC2 instance and use it to process some data and have SQLite on EC2 root volume which solves the issue
s

Szymon Szyszko

11/03/2022, 12:42 PM
I run prefect based images in Kubernetes as k8s jobs. Without shared storage like EFS, orion.db would be lost after k8s job termination. I mentioned EC2 because it is simpler to reproduce the problem on EC2 and logs are the same.
It would work with Postgres. For some reasons, I would like to try with Sqlite
a

Anna Geller

11/03/2022, 2:54 PM
I'd strongly recommend Postgres or Prefect Cloud for your use case, sqlite is only if you run everything on one machine e.g. your laptop
for self-hosted setup on K8s, check https://github.com/PrefectHQ/prefect-helm