i'm a new user i want to save the flows to the database PostgreSQL, I added the connection and the p...

Rabea Yousof

10/30/2022, 10:41 AM

i'm a new user i want to save the flows to the database PostgreSQL, I added the connection and the prefect tables was added to my database, I have a record in table configuration , But the flows and the logs not inserted to my PostgreSQL database, i don’t know what i’m missing, can someone help me please?

✅ 1

Anna Geller

10/30/2022, 11:38 AM

Welcome, @Rabea Yousof! You don't need to store your flows to Postgres, we use the concept of storage for that. If you run everything locally, you don't even have to worry about storage. Check out this PDF for details on deployments and this repo this might be also good to get started https://medium.com/the-prefect-blog/scheduled-data-pipelines-in-5-minutes-with-prefect-and-github-actions-39a5e4ab03f4

Rabea Yousof

10/30/2022, 12:01 PM

Thank you, i need to apply this feature with PostgreSQL to keep the data on my database

Anna Geller

10/30/2022, 2:02 PM

what feature? what problem are you trying to solve?

Rabea Yousof

10/30/2022, 2:07 PM

i'm trying to save the the flows and tasks history and logs to PostgreSQL https://docs.prefect.io/concepts/database/#configuring-a-postgresql-database i applied the all steps, but when i run the flow the data not saved to the prefect tables in the DB

Anna Geller

10/30/2022, 2:12 PM

it happens by default, no need to do anything there, the history and logs are written to DB

Anna Geller

10/30/2022, 2:12 PM

if you instead mean task and flow run results, you can set:

Copy code

prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT=true

Anna Geller

10/30/2022, 2:13 PM

this way all results will be by default persisted in your default storage location

Zanie

11/01/2022, 3:08 PM

It sounds like you’re trying to use the Postgres database backend instead of SQLIte? Did you resolve this?

Zanie

11/01/2022, 3:08 PM

You’ll want to use

prefect config set PREFECT_ORION_DATABASE_CONNECTION_URL="postgresql+asyncpg://..."

so that it applies to all your flows

Zanie

11/01/2022, 3:09 PM

@terrence Maybe the

export

in that tutorial should be a

prefect config set

👍 1

Rabea Yousof

11/02/2022, 7:46 PM

@Zanie Yes, i'm using Postgres as backend now with the local prefect and it's working very good. Can i use Postgres database backend or secondary database with prefect cloud?

Zanie

11/02/2022, 8:10 PM

Prefect Cloud has its own backing database. You can use profiles to switch between local/cloud if you want, but they wont share data.

👍 1

Rabea Yousof

11/07/2022, 10:08 AM

Hi @Zanie, can you help me with this error, I added the connection string and i started the Orion,

Matthew Jeffs

12/01/2022, 6:48 AM

@Rabea Yousof @Zanie were you able to get this resolved. I recently switched from SQLite to Postgres and got that error. I am working to try and troubleshoot now.

Zanie

12/01/2022, 6:56 AM

I think we’ll need a full bug report at https://github.com/PrefectHQ/prefect/issues/new?assignees=&labels=status%3Atriage%2Cbug&template=1_general_bug_report.yaml to help with that

20 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.