Madison Schott

    Madison Schott

    4 months ago
    Hi 🙂 wondering if anyone has resources for creating a (near) real-time data pipeline using Fivetran and Prefect?
    Kevin Kho

    Kevin Kho

    4 months ago
    I dont think you can do this in Prefect 1 because the minimal schedule is 1 minute. If that is close enough, then it should just be a matter of calling the FiveTranSync task at that interval right?
    Madison Schott

    Madison Schott

    4 months ago
    yeah that would be fine, I think the minimum default sync time for Fivetran is actually 15min
    I guess how would you be sure the next sync doesn't run for a connector without the first one finishing?
    Kevin Kho

    Kevin Kho

    4 months ago
    You can use a Flow Run concurrency limit or right a state handler to exit the second flow if it already sees a running one