Hi everyone, I'm interested in using prefect to continuously replicate a bigquery table into a postgres database. Is this possible to do within prefect?

Hi @George Tam! Are you asking more about the continuous part of the bigquery to postgres part? or both?

both, can prefect handle that type of extract and load job?

If continuous streaming, not really unfortunately. If it’s like an hourly process (or even 5 min process), yes. Bigquery to postgres, if you can do it in Python (which i think it can be), then it can be done.

yea hourly/ 5 min job is fine, do you have any documentation for this type of job? just learned about prefect a few hours ago and curious thank you

Unfortunately, that’s a specific use case I don’t have documentation for but what I do have to show is the task library . The is already a BigQuery task, as well as a Postgres Task. There is also a small snippet how to use tasks in that page a bit below. If you have anymore specific questions, you can feel free to ask them here and we’ll respond to you

thanks