Can I have multiple overlapping instances of a given flow run (let’s saw with different parameters)? If so can I have shared state between those runs (preferable in memory rather than on disk)?
01/13/2021, 10:14 PM
Hi @Joseph multiple flow runs do not have the option to share state. Is there a specific use case you have in mind?
01/14/2021, 1:37 AM
Thanks for the reply. I have a flow which creates jobs that populate partitions of a database. Every time that the flow is run, it may discover new partitions. It is possible that one instance of the Flow will still be running (populating one partition, which can take hours) while the next instance of the Flow begins running. I need some state to track which partitions are: not yet populated, still being populated and have had a failure during population.
The two ways I could imagine this being modeled in Prefect is either as a flow which is continuously runs (conceptually in a loop) or as many invocations of the same flow, some instances of which may overlap in run time.