Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi :slightly_smiling_face: wondering if anyone has resources for creating a (near) real-time data pipeline using Fivetran and Prefect?

I dont think you can do this in Prefect 1 because the minimal schedule is 1 minute. If that is close enough, then it should just be a matter of calling the FiveTranSync task at that interval right?

yeah that would be fine, I think the minimum default sync time for Fivetran is actually 15min

I guess how would you be sure the next sync doesn't run for a connector without the first one finishing?

You can use a Flow Run concurrency <https://docs.prefect.io/orchestration/flow-runs/concurrency-limits.html#flow-run-limits|limit> or right a state handler to exit the second flow if it already sees a running one