I'm speaking in hypotheticals here, I haven't dug ...
# prefect-communityk
Kevin Hill
12/06/2019, 6:05 PMI'm speaking in hypotheticals here, I haven't dug entirely through the framework (minus the easy first example), though I wanted to ask a few questions:
1. Can if I had a hypothetical Reinforcement Learning training pipeline run through prefect, could it handle millions of interactions over the course of a training period?
2. Is it possible to push information from the end of the flow to the beginning if a condition is met?
3. If so, how fast would that be? Can I use Dask to split certain tasks up between machines and concurrently process more tasks?
4. Would that use the
LOOPKevin Hill
12/06/2019, 6:38 PMWhen I say push from end to start, I mean something like the
.connectstreamzd
Dylan
12/06/2019, 10:20 PM
Hey Kevin! Prefect Core + Prefect Scheduler would definitely solve your usecase. I’ll run through what I’d suggest
Dylan
12/06/2019, 10:21 PM
First, I’d set up a Dask Cluster and configure your flows to run on that cluster using our
RemoteEnivronmentDylan
12/06/2019, 10:21 PM
I’d set your flow to run with
ParametersDylan
12/06/2019, 10:22 PM
As the last task in your flow, if the condition is met, I would kick off a new run of your flow using the Prefect Scheduler GraphQL API via the python client
Dylan
12/06/2019, 10:22 PM
Your hypothetical Reinforcement Learning training pipeline would handle millions of interactions over the course of a training period
Dylan
12/06/2019, 10:23 PM
and you could run an arbitrary number of flows on the same infrastructure
Dylan
12/06/2019, 10:23 PM
Let me know if this helps @Kevin Hill!
Dylan
12/06/2019, 10:23 PM
Here’s some supporting documentation:
Dylan
12/06/2019, 10:24 PM
Dylan
12/06/2019, 10:24 PM
Dylan
12/06/2019, 10:25 PM