Hey guys,
We are running a few pipelines on Prefect 1.2 (on-prem Kubernetes) and recently i've started working on migrating to Orion.
We have two use cases that query Prometheus metrics: streaming flows running at seconds interval and batch flows running for the previous day, weeks etc.
In Prefect 1.2 I had two projects, one for streaming and one for batch. In this way, I was able to register a flow having the same name in both projects (of course, having different tasks under the hood). For example, i have a flow called 'Alerts' in both projects, one running daily and fetching previous day data and another one running at 30 seconds interval.
In Orion, it seems it's not possible anymore to isolate the two use cases. Is there another way? I've read about Workspaces, but that it's available only in Prefect Cloud.
Another question is about starting streaming flows. My flow consists in having one task creating a table and another one running a while loop that calls the Prometheus api every 30 seconds and inserts data into the table. So i need to run in streaming mode just one task (the api call), not the entire flow. In Prefect 1.2 i am scheduling the flow to run only once 10 seconds after the registration (using
from prefect.schedules import Schedule
). In this way, it's entering the task while loop and the UI shows only one flow run in "Running" state (until you decide to kill it).
Is there a way to replicate this scenario in Orion? If i use the Interval schedule, it's creating flow runs every 30 seconds and it's difficult for the users to read the UI once you have more flows running.
Thank you!