https://prefect.io logo
l

Leo Meyerovich (Graphistry)

10/18/2020, 12:19 AM
We're trying to figure out how to dispatch several concurrent interval flows, but the docs bottom out on this. Any pointers? 1. Original:
for p in configs: interval_flow.run(parameters=p)
<= was not concurrent 2. Docs: flow docs say to manually call `FlowRunner`: `for p in configs: FlowRunner(flow=interval_flow).run()`<= unclear if the intervals run concurrently, and unclear how to pass in different
p
to different flows Any pointers?
n

nicholas

10/19/2020, 2:41 PM
Hi @Leo Meyerovich (Graphistry) - just to confirm: you're running Core-only, not using Prefect Server or Prefect Cloud, correct?
l

Leo Meyerovich (Graphistry)

10/19/2020, 4:15 PM
Not sure of terminology -- most immediately, we're back to launch direct local flows (
pip install prefect
=>
.py
with local
Flow
), and once we're smooth on single node, starting again on what is/was multiple
prefect agent
supervisord's /
backend server
with remote clients calling them. (when we initially used prefect, it was rough going relative to vs. not, so starting up again.)
n

nicholas

10/19/2020, 5:09 PM
@Leo Meyerovich (Graphistry) if you're directly calling
flow.run
, you're only running things in process; to run flows concurrently you'll need a persistent backend (like Prefect Server), which you can use to start runs using the
create_flow_run
mutation from another client (or you can use the UI to start them). You can read more about Prefect Server here. If you'd rather not host your own version of Server, you can sign up for a free Prefect Cloud account instead,
l

Leo Meyerovich (Graphistry)

10/19/2020, 5:35 PM
Yes, I assume we'll be running Server (
backend server
that we were using before?) I'm still unclear of the answer to the original question, even assuming server. Imagine jupyter notebooks where an analyst triggers a notebook that should launch multiple Interval (on-going) flows, where `.run()`'s blocking gets stuck on the first flow, and the doc's recommended
FlowRunner()
alternative has no clear way to pass a parameter..
My mental model is
FlowRunner
is the right choice as (I hope) it'd enable a detached task so that the notebook can spawn multiple flows and exit, and the interval schedule keeps ticking w/out needing the notebook to be running. But still learning, e.g., can't get
FlowRunner
to pass a parameter..
n

nicholas

10/19/2020, 5:40 PM
Got it @Leo Meyerovich (Graphistry) - I think you'll need to call
FlowRunner.run
, to which you can pass parameters. You can also call
FlowRunner.initialize_run
for finer-control of that.
However, I believe that will directly bypass any schedule you set so I'm not sure about the
Interval
portion of your question
c

Chris White

10/19/2020, 5:46 PM
I’m a little confused on this thread - using a Prefect backend handles all of this. - register a flow with an interval schedule, the scheduler creates flow runs - one or many agents submit these flow runs for concurrent execution - parameters can be passed to ad-hoc created flow runs, they can be passed on each individual Clock for scheduled runs, or they can be updated via the Flow’s settings tab in the UI Using a FlowRunner directly is only something you should do during testing, and as nicholas mentioned, will bypass any schedule you may have on your flow
l

Leo Meyerovich (Graphistry)

10/19/2020, 5:50 PM
Can an agent do:
Copy code
for p in configs:
    make_interval_flow_instance(p).run()
The docs make it sound like the first submitted interval must run to completion before next gets submitted.
Testing locally makes it seem the case that
.run()
is blocking; we were trying to make that work before getting around
backend server
running again
c

Chris White

10/19/2020, 5:50 PM
I’m sorry I’m not following what you’re asking; agents submit runs concurrently, there is no blocking call within agents
l

Leo Meyerovich (Graphistry)

10/19/2020, 5:51 PM
flow.run()
is not blocking?
c

Chris White

10/19/2020, 5:53 PM
yes that is a blocking call, but that method call is not used by Agents.
l

Leo Meyerovich (Graphistry)

10/19/2020, 5:55 PM
so if we want one python notebook to submit multiple interval jobs (and ideally have them run in a state detatched from the notebook kernel), is there some other call / doc I should be looking at?
Sounds like I'm very much in the wrong place, apologies 🙂
c

Chris White

10/19/2020, 6:01 PM
So let me use Prefect terminology, because I’m not sure what you mean when you say an “interval job”: - you can register your Flows (which can include their interval schedule) from within a notebook and then shut that notebook down; this will activate the scheduler which will begin scheduling runs for your registered flows until the
end_date
for the schedule is reached - when the scheduled start times of your runs approaches, your agent(s) will submit the run for execution. Agents will submit as many runs as they find, there are no blocking calls or anything. It might be useful to walk through our getting started guides: - https://docs.prefect.io/orchestration/tutorial/first.html - https://docs.prefect.io/orchestration/execution/overview.html
🎩 1
l

Leo Meyerovich (Graphistry)

10/19/2020, 6:03 PM
Thanks, will start over w/ a backend server. I think you're right in the original diagnosis that trying to go far w/ local /backend-less mode was confusing this.
👍 1
(In particular, sounds like
.register()
means no need for
.run()
gymnastics)
2 Views