Hello guys, I have a weird behaviour while using prefect Orion deployments with the UI, I have this deployment file:

import pandas as pd
from prefect.deployments import DeploymentSpec
from prefect.orion.schemas.schedules import IntervalSchedule

from pred.predict import predict_flow

DeploymentSpec(
    flow=predict_flow,
    name="Prediction"
)

I create the deployment without a problem and I can see it in the UI. I also can run it successfully from CLI using:

prefect deployment execute 'Prediction flow/Prediction'

But When I try to run it using the “Quick Run” in the UI, a “Scheduled” task is created and is never run. I can see it in the lateness graph. one task didn’t run for over an hour during my lunch break. Same happens if I add a schedule to the deployment. any ideas why that happens or how to fix it?

Just yesterday, there was a new Orion release which makes the process way easier thanks to FlowRunners. Can you upgrade your Orion version (

pip install -U "prefect>=2.0a6"

) and try e.g. this tutorial? It shows how deployment can be run on an agent. There is also this brand new concept documentation of Deployments that can help understand how this works.

Thank you Anna! Upgrading prefect to “2.0a6” solved the “Quick Run” problem. I’m testing it now with a schedule. I’ll post again if there is still a problem

It only means that Orion can’t schedule flow runs specified in your Deployment unless the Orion instance with scheduler and agent services is started. This means that you can create deployments at any time, but in order for Orion to schedule flow runs, the Orion instance must be running. When you create deployments, you only send metadata to Orion’s backend so that it knows what flow to run, how to run it (parameters), when to run it (schedule) and where to run it (flow runner). The actual runs are handled by the services you spawn when you start Orion. Regarding the last question, I think this is what the late runs service is for, your scheduled runs should be picked up by the scheduler and submitted to the agent once you restarted Orion.

By “stateful Orion services” I mean running

prefect orion start

The reason I’m asking is that when I try to create a deployment without running

prefect orion start

first, The flow runs are still created but they never run as you see from the appended screenshot. The only way I got them to work is by making sure

prefect orion start

runs first before creating any deployment.

Gotcha, that seems to make sense though, right? because before you run “prefect orion start”, there is no agent to pick up your flow run, so UI is correct to display those as scheduled and basically waiting. Or would you expect a different behavior?

It’s ok to have this behaviour I guess, my question then would be: Is there a way to make an agent pick up those flow runs later?

I think once you would deploy your flows to “production”, then it shouldn’t be an issue since the agent will be up at all times, and I think it’s possible that you could even have two agents so that a separate agent service can pick up flow runs if the main agent service is down somehow. And my understanding is that the

MarkLateRuns

 service places such flow runs (not picked up by an agent at the time they were schedule to run) in a 

Late

 state so that they can be picked up for execution later.

Maybe this

MarkLateRuns

doesn’t run efficiently as I expect then. I also can’t see a way in documentation on how can we make sure that such service is up and running as expected, or see the flows is detected. I’m not sure I fully understand everything yet but I guess I have enough information to keep me going. Thanks!