Pedro Machado
05/23/2023, 8:10 PMcheck_data_availability
task that calls the API. If the data is not ready, it should raise an exception.
• If the data is ready, it will continue
Can you think of a better pattern? If I go with this approach, I was thinking that it would be good to define a new state waiting for data availability
or something along those lines to differentiate it from a failure. Does this make sense? Are custom states possible? Can Prefect Cloud render custom states? How would I specify a custom state?
Thanks!Henning Holgersen
05/23/2023, 8:27 PMNate
05/23/2023, 8:32 PMwaiting for data availability
when data is not ready, in that case you could probably just call run_deployment
with a scheduled_time
in the future - that way you're not spending runtime just waiting (as Henning notes above)
that way you could just schedule it to run once a month on the 6th, and let it reschedule itself every so often until it can get the data and complete
note that ideally, you can event off the availability of the data directly, so that some external system knows that your data is ready and can just call create_flow_run_from_deployment
endpoint whenever that happensPedro Machado
05/23/2023, 8:38 PMrun_deployment
with a future date as an alternative!