Leo Meyerovich (Graphistry)10/18/2020, 12:19 AM
<= was not concurrent 2. Docs: flow docs say to manually call `FlowRunner`: `for p in configs: FlowRunner(flow=interval_flow).run()`<= unclear if the intervals run concurrently, and unclear how to pass in different
for p in configs: interval_flow.run(parameters=p)
to different flows Any pointers?
Leo Meyerovich (Graphistry)10/19/2020, 4:15 PM
pip install prefect
), and once we're smooth on single node, starting again on what is/was multiple
with remote clients calling them. (when we initially used prefect, it was rough going relative to vs. not, so starting up again.)
, you're only running things in process; to run flows concurrently you'll need a persistent backend (like Prefect Server), which you can use to start runs using the
mutation from another client (or you can use the UI to start them). You can read more about Prefect Server here. If you'd rather not host your own version of Server, you can sign up for a free Prefect Cloud account instead,
Leo Meyerovich (Graphistry)10/19/2020, 5:35 PM
that we were using before?) I'm still unclear of the answer to the original question, even assuming server. Imagine jupyter notebooks where an analyst triggers a notebook that should launch multiple Interval (on-going) flows, where `.run()`'s blocking gets stuck on the first flow, and the doc's recommended
alternative has no clear way to pass a parameter..
is the right choice as (I hope) it'd enable a detached task so that the notebook can spawn multiple flows and exit, and the interval schedule keeps ticking w/out needing the notebook to be running. But still learning, e.g., can't get
to pass a parameter..
, to which you can pass parameters. You can also call
for finer-control of that.
portion of your question
Leo Meyerovich (Graphistry)10/19/2020, 5:50 PM
The docs make it sound like the first submitted interval must run to completion before next gets submitted.
for p in configs: make_interval_flow_instance(p).run()
is blocking; we were trying to make that work before getting around
Leo Meyerovich (Graphistry)10/19/2020, 5:51 PM
is not blocking?
Leo Meyerovich (Graphistry)10/19/2020, 5:55 PM
for the schedule is reached - when the scheduled start times of your runs approaches, your agent(s) will submit the run for execution. Agents will submit as many runs as they find, there are no blocking calls or anything. It might be useful to walk through our getting started guides: - https://docs.prefect.io/orchestration/tutorial/first.html - https://docs.prefect.io/orchestration/execution/overview.html
Leo Meyerovich (Graphistry)10/19/2020, 6:03 PM
means no need for