https://prefect.io logo
Title
m

Matt Harvey

10/22/2019, 7:47 PM
I'm trying to decide on best practices for a complex flow schedule. I have a flow that needs to run at specific times each day, but those times change each day depending on data obtained from an API in the morning. Before I moved this flow to Prefect and did it the old fashion way, I just ran the script every minute, and it started with a function call that returned a bool depending on if it should continue or not. (That function would do the complex logic of hitting the API or retrieved data and determine if it should run at that moment). In Prefect, should I create a task that does this check and then skips the rest of the flow if it is not supposed to run? Or is there a way I can create a dynamic schedule that I pass into the flow itself? Or is there a better way altogether?
j

Jeremiah

10/22/2019, 8:48 PM
@Matt Harvey that’s an excellent question. In Prefect, we think of schedules as a stand-in for manually starting your flow — in other words, just a convenience for using the API to create flow runs with known future start times. In Prefect Cloud, I think we’d actually suggest not using a schedule at all for this Flow (because the future start times aren’t well known). Instead, we’d recommend creating a small flow that hits that API on a regular basis and then creates new runs of your main flow with appropriate
start_time
settings. For example, you might hit the API at 9am every morning and create 5 new runs of the main flow in response.
The more traditional approach is exactly what you suggested — run the flow more frequently and use a task to check whether the entire run should skip. This is completely valid pattern; the pattern I suggested above is just how we would take advantage of the Cloud API.
m

Matt Harvey

10/22/2019, 9:07 PM
@Jeremiah Thanks for the response. Creating new runs with the appropriate start times feels like a clean solution for my use case because although start times are not known for a given day, I do know when the list of start times becomes available each morning. Would you be able to point me towards the pertinent documentation for creating flow runs on the fly like that?
j

Jeremiah

10/22/2019, 9:16 PM
@Matt Harvey our documentation on the Cloud API is admittedly lacking at the moment and will get an upgrade soon. However, the GraphQL call is documented here (https://docs.prefect.io/cloud/concepts/flow_runs.html#scheduling-a-flow-run) and the equivalent call from the Python client is documented here (https://docs.prefect.io/api/unreleased/client/client.html#prefect-client-client-client-create-flow-run)
If we can provide assistance with getting any of this set up in the interim, please ping either me or @Chris White (or both)!
m

Matt Harvey

10/22/2019, 9:36 PM
Thanks as always. Prefect is a treat to work with.
:marvin: 1