Hi everyone 👋 My goal is to put Sqlite orion.db in EFS mounted into EC2 instance. It generates an error. Error logs are attached in the thread to keep the channel clean. According to Amazon Elastic File System (Amazon EFS) Now Supports NFSv4 Lock Upgrading and Downgrading it is possible to use Sqlite in EFS. I noticed that in src/prefect/orion/database/configurations.py#L262 you set journal_mode to

WAL

which is not allowed in network file systems. I changed that to

delete

without success - logs do not change. I have seen similar logs to mine here In my project, I have other Sqlite databases in the EFS directory and they work fine.

Have you tried those suggestions? it might be a little easier https://prefect-community.slack.com/archives/CL09KU1K7/p1662771005862079?thread_ts=1662769580.637409&cid=CL09KU1K7

No. I would rather find source of the problem than workaround. I did not use Docker to reproduce the problem. I run Python code on EC2 with EFS mounted to /home/ubuntu/.prefect Definitely using EFS causes the problem. When EFS is not mounted it works.

Using SQLite backend is meant more for a lightweight setup (no mounts), and if you need something more robust or bigger scale, you would either use PostgreSQL or point your profile to Prefect Cloud.

I do not need a bigger scale. SQLite fits my load needs. I don't know how placing SQLite in network mount makes a solution not lightweight. It can but in my case, it will not.

can you tell me why you want to put SQLite on EFS, then? you can still mount an EFS volume to your EC2 instance and use it to process some data and have SQLite on EC2 root volume which solves the issue

I run prefect based images in Kubernetes as k8s jobs. Without shared storage like EFS, orion.db would be lost after k8s job termination. I mentioned EC2 because it is simpler to reproduce the problem on EC2 and logs are the same.

I'd strongly recommend Postgres or Prefect Cloud for your use case, sqlite is only if you run everything on one machine e.g. your laptop