Is there a best practice for getting a flow to run continuously? So it starts over again after the last task is completed..
j
josh
04/01/2020, 6:27 PM
Off the top of my head the way I would implement it would be to have the last task of the flow schedule another run of the same flow using something like this: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/tasks/cloud/flow_run.py#L9
Or I would set the flow to have a schedule which runs frequently. But this is dependent on if you can accurately predict how long a run of the flow will take
d
David N
04/01/2020, 6:28 PM
I cannot accurately say, so will look at your first suggestion. Thanks!!
David N
04/02/2020, 12:17 AM
i think thats only for cloud, if you happen to have core suggestions, Im all ears.
j
josh
04/02/2020, 1:22 AM
Are you using Prefect core’s server? If so we could update this task (or add another one) to opt into the cloud project piece 🙂
d
David N
04/02/2020, 6:16 PM
I appreciate you trying to help me, but I dont really understand, sorry. We're not using the server currently, I kinda thought that was only for the UI which isn't accessible for us (we're hosting our prefect env in ec2)
David N
04/02/2020, 6:17 PM
in the meantime ive written an sh script that checks if our workflow is running every minute and kicks it off again.. not ideal but it works for now
j
josh
04/02/2020, 6:17 PM
Okay glad it works! Yeah outside of a frequently running schedule or a state persistence layer I think some external process might be the best way to go
d
David N
04/02/2020, 6:26 PM
Thanks again. I see some event based triggers on the road map. If I understand correctly, maybe a final task that raises an event, which starts another run of the same flow will work when thats available.