Andrey Alekseev
06/18/2020, 8:40 AMemre
06/18/2020, 12:57 PMfeature
and predict
tasks and chain them together, effectively unrolling the loop.
Or, you can combine the generation and prediction tasks, and loop the task using the LOOP
signal.
Personally, I prefer task looping. Given how co-dependent feature
and predict
are, separating them isn’t worth the trouble.
https://docs.prefect.io/core/examples/task_looping.htmlJeremiah
Andrey Alekseev
06/18/2020, 1:18 PMemre
06/18/2020, 1:18 PMJeremiah
Flow
and you could have a task kick off those external flows via API as many times as you want. This would essentially amount to a small Python function looping continuously inside Orchestrator-Flow
that kicks off the Feature-and-Train
flow with a parameter containing the data via API, retrieving its result (also via API) and looping appropriately. You would get all the benefits of Prefect introspection and retries, while also creating a cyclical dependency. I wouldn’t say this is something we see especially often (@emre’s approach is more common) but whatever meets your unit of work is the best option.Andrey Alekseev
06/18/2020, 2:20 PM