Leo Meyerovich (Graphistry)
10/18/2020, 12:19 AMfor p in configs: interval_flow.run(parameters=p)
<= was not concurrent
2. Docs: flow docs say to manually call `FlowRunner`: `for p in configs: FlowRunner(flow=interval_flow).run()`<= unclear if the intervals run concurrently, and unclear how to pass in different p
to different flows
Any pointers?nicholas
Leo Meyerovich (Graphistry)
10/19/2020, 4:15 PMpip install prefect
=> .py
with local Flow
), and once we're smooth on single node, starting again on what is/was multiple prefect agent
supervisord's / backend server
with remote clients calling them. (when we initially used prefect, it was rough going relative to vs. not, so starting up again.)nicholas
flow.run
, you're only running things in process; to run flows concurrently you'll need a persistent backend (like Prefect Server), which you can use to start runs using the create_flow_run
mutation from another client (or you can use the UI to start them). You can read more about Prefect Server here. If you'd rather not host your own version of Server, you can sign up for a free Prefect Cloud account instead,Leo Meyerovich (Graphistry)
10/19/2020, 5:35 PMbackend server
that we were using before?)
I'm still unclear of the answer to the original question, even assuming server. Imagine jupyter notebooks where an analyst triggers a notebook that should launch multiple Interval (on-going) flows, where `.run()`'s blocking gets stuck on the first flow, and the doc's recommended FlowRunner()
alternative has no clear way to pass a parameter..FlowRunner
is the right choice as (I hope) it'd enable a detached task so that the notebook can spawn multiple flows and exit, and the interval schedule keeps ticking w/out needing the notebook to be running. But still learning, e.g., can't get FlowRunner
to pass a parameter..nicholas
FlowRunner.run
, to which you can pass parameters. You can also call FlowRunner.initialize_run
for finer-control of that.Interval
portion of your questionChris White
Leo Meyerovich (Graphistry)
10/19/2020, 5:50 PMfor p in configs:
make_interval_flow_instance(p).run()
The docs make it sound like the first submitted interval must run to completion before next gets submitted..run()
is blocking; we were trying to make that work before getting around backend server
running againChris White
Leo Meyerovich (Graphistry)
10/19/2020, 5:51 PMflow.run()
is not blocking?Chris White
Leo Meyerovich (Graphistry)
10/19/2020, 5:55 PMChris White
end_date
for the schedule is reached
- when the scheduled start times of your runs approaches, your agent(s) will submit the run for execution. Agents will submit as many runs as they find, there are no blocking calls or anything.
It might be useful to walk through our getting started guides:
- https://docs.prefect.io/orchestration/tutorial/first.html
- https://docs.prefect.io/orchestration/execution/overview.htmlLeo Meyerovich (Graphistry)
10/19/2020, 6:03 PM.register()
means no need for .run()
gymnastics)