d

    David Stern

    2 years ago
    Apologies if this has already been answered, but the docs don't make it super clear what terminology I should use so it's difficult to search for an answer. Basically, I want to know if we can have a flow where it waits for one task to complete, basically because some data isn't yet available from a provider, so it checks that for example every 15 mins and continues the flow when it's available? And importantly, is this available in the open-source version? This is the crucial reason we'd use Prefect instead of exclusively using something like
    dbt
    , which I think we'll use anyway.
    Chris White

    Chris White

    2 years ago
    Hi David - definitely. Here’s what I’d suggest:
    from datetime import timedelta
    
    @task(max_retries=500, retry_delay=timedelta(minutes=15))
    def check_data_available():
        # hit the provider to determine whether
        # data is available
        if data_available:
            return None # can optionally return some info
        else:
            raise RuntimeError("Data not yet available.")
    d

    David Stern

    2 years ago
    Thanks so much Chris!