https://prefect.io logo
Title
t

Toby Rahloff

03/09/2022, 8:21 AM
Hi, we are currently evaluation Prefect Orion (and we are pretty amazed to be honest). One challenge I cannot wrap my had around at the moment is manual interaction (e.g., "wait for approval"). We are crunching large visual data sets that often have unexpected features. Because of this, we need some kind of manual sanity check by a human expert before we start the compute/time intensive tasks. In Prefect 1.0 this was possible via the
manual_only
trigger. Is it possible to do the same with Orion (in the future)?
a

Anna Geller

03/09/2022, 9:13 AM
As of today, there are no Paused or Resume states, so this isn't possible in Orion yet. Still, I'll open an issue internally to discuss with other engineers whether (and how) this could be added. For now, perhaps you could build this manual approval as part of your CI/CD pipeline? Many CI/CD tools have such functionality (even though this would likely be on a flow/subflow level then).
t

Toby Rahloff

03/09/2022, 9:41 AM
Nice, thanks for the quick response! So the best-practice approach suggest by Prefect would be to model a pipeline as two different deployments and a CI/CD pipeline to orchestrate deployment runs (and the manual approval between both runs)? I could kinda see that work. More generally speaking: Do you have a recommended approach in Prefect Orion for pipeline triggers/orchestration besides manually triggering deployment runs?
a

Anna Geller

03/09/2022, 10:11 AM
I think we are not at the stage to make best-practice recommendations for Orion yet 🙂 Orion is still in a technical preview and is under very active development. The CI/CD option I shared is more a workaround until we have similar functionality in Orion for handling such manual approval use cases. Regarding your second question, you can attach a schedule to your
DeploymentSpec
to run it on a given cadence:
from prefect.deployments import DeploymentSpec
from prefect.orion.schemas.schedules import IntervalSchedule
from datetime import timedelta

# note that deployment names are 
# stored and referenced as '<flow name>/<deployment name>'
DeploymentSpec(
    flow_location="/Developer/workflows/my_flow.py",
    name="my-first-deployment",
    parameters={"nums": [1, 2, 3, 4]}, 
   schedule=IntervalSchedule(interval=timedelta(minutes=15)),
    
)
🙌 1