We're trying to figure out how to dispatch several concurrent interval flows, but the docs bottom out on this. Any pointers? 1. Original:

for p in configs: interval_flow.run(parameters=p)

<= was not concurrent 2. Docs: flow docs say to manually call `FlowRunner`: `for p in configs: FlowRunner(flow=interval_flow).run()`<= unclear if the intervals run concurrently, and unclear how to pass in different

p

to different flows Any pointers?

Hi @Leo Meyerovich (Graphistry) - just to confirm: you're running Core-only, not using Prefect Server or Prefect Cloud, correct?

Not sure of terminology -- most immediately, we're back to launch direct local flows (

pip install prefect

=>

.py

with local

Flow

), and once we're smooth on single node, starting again on what is/was multiple

prefect agent

supervisord's /

backend server

with remote clients calling them. (when we initially used prefect, it was rough going relative to vs. not, so starting up again.)

@Leo Meyerovich (Graphistry) if you're directly calling

flow.run

, you're only running things in process; to run flows concurrently you'll need a persistent backend (like Prefect Server), which you can use to start runs using the

create_flow_run

mutation from another client (or you can use the UI to start them). You can read more about Prefect Server here. If you'd rather not host your own version of Server, you can sign up for a free Prefect Cloud account instead,

Yes, I assume we'll be running Server (

backend server

that we were using before?) I'm still unclear of the answer to the original question, even assuming server. Imagine jupyter notebooks where an analyst triggers a notebook that should launch multiple Interval (on-going) flows, where `.run()`'s blocking gets stuck on the first flow, and the doc's recommended

FlowRunner()

alternative has no clear way to pass a parameter..

Got it @Leo Meyerovich (Graphistry) - I think you'll need to call

FlowRunner.run

, to which you can pass parameters. You can also call

FlowRunner.initialize_run

for finer-control of that.

I’m a little confused on this thread - using a Prefect backend handles all of this. - register a flow with an interval schedule, the scheduler creates flow runs - one or many agents submit these flow runs for concurrent execution - parameters can be passed to ad-hoc created flow runs, they can be passed on each individual Clock for scheduled runs, or they can be updated via the Flow’s settings tab in the UI Using a FlowRunner directly is only something you should do during testing, and as nicholas mentioned, will bypass any schedule you may have on your flow

Can an agent do:

for p in configs:
    make_interval_flow_instance(p).run()

The docs make it sound like the first submitted interval must run to completion before next gets submitted.

I’m sorry I’m not following what you’re asking; agents submit runs concurrently, there is no blocking call within agents

flow.run()

is not blocking?

yes that is a blocking call, but that method call is not used by Agents.

so if we want one python notebook to submit multiple interval jobs (and ideally have them run in a state detatched from the notebook kernel), is there some other call / doc I should be looking at?

So let me use Prefect terminology, because I’m not sure what you mean when you say an “interval job”: - you can register your Flows (which can include their interval schedule) from within a notebook and then shut that notebook down; this will activate the scheduler which will begin scheduling runs for your registered flows until the

end_date

for the schedule is reached - when the scheduled start times of your runs approaches, your agent(s) will submit the run for execution. Agents will submit as many runs as they find, there are no blocking calls or anything. It might be useful to walk through our getting started guides: - https://docs.prefect.io/orchestration/tutorial/first.html - https://docs.prefect.io/orchestration/execution/overview.html

Thanks, will start over w/ a backend server. I think you're right in the original diagnosis that trying to go far w/ local /backend-less mode was confusing this.