https://prefect.io logo
Title
m

Madison Schott

05/24/2022, 4:50 PM
Hi 🙂 wondering if anyone has resources for creating a (near) real-time data pipeline using Fivetran and Prefect?
k

Kevin Kho

05/24/2022, 4:59 PM
I dont think you can do this in Prefect 1 because the minimal schedule is 1 minute. If that is close enough, then it should just be a matter of calling the FiveTranSync task at that interval right?
m

Madison Schott

05/24/2022, 5:04 PM
yeah that would be fine, I think the minimum default sync time for Fivetran is actually 15min
I guess how would you be sure the next sync doesn't run for a connector without the first one finishing?
k

Kevin Kho

05/24/2022, 5:06 PM
You can use a Flow Run concurrency limit or right a state handler to exit the second flow if it already sees a running one