https://prefect.io logo
Title
b

Bob Colner

03/26/2020, 5:20 AM
Question about migrating to prefect cloud. How can I get multiple agent to 'listen' to my cloud flow for task parallelism? This https://docs.prefect.io/cloud/tutorial/multiple.html#run-an-agent-manually talks about agents running multiple flows but what about multiple tasks?
can a agent be multi-threaded?
c

Chris White

03/26/2020, 5:26 AM
Hi Bob, Agents don’t run or submit tasks, they only submit Flow runs. Executors are the Prefect objects responsible for submitting tasks for execution. We have two dask-based executors for parallel task execution which can be found here: https://docs.prefect.io/api/latest/engine/executors.html For multithreading, I recommend using
LocalDaskExecutor(schduler="threads")
. This can be configured for your Flow runs in Cloud through an execution environment, as described here: https://docs.prefect.io/cloud/execution/remote_environment.html#overview
Apologies for the information overload! I highly recommend experimenting with the different executors locally using Core before experimenting with the environments in Cloud - let us know if you run into any issues or have any additional questions!
b

Bob Colner

03/26/2020, 3:56 PM
thanks, I am strugging with so many concept and the docs don't give a coherent overview.
👍 1
c

Chris White

03/26/2020, 3:57 PM
Sounds like you would benefit from walking through our tutorial: https://docs.prefect.io/core/tutorial/01-etl-before-prefect.html
b

Bob Colner

03/26/2020, 3:58 PM
I'm using
LocalDaskExecutor(schduler="threads")
to run locally but I don't see anywhere in the docs about . how to use that with the cloud service
c

Chris White

03/26/2020, 3:59 PM
Check out the remote environment docs I linked to above; the
RemoteEnvironment
allows you to configure an executor for your Cloud flow runs; in particular this section can be tweaked to support the `LocalDaskExecutor`: https://docs.prefect.io/cloud/execution/remote_environment.html#examples
b

Bob Colner

03/26/2020, 3:59 PM
I've looked at the tutorials, they each let you understand a single issue / use-case but dont really let me see how all the pieces can to used together.
c

Chris White

03/26/2020, 4:00 PM
Prefect was designed to be like a collection of lego blocks, where each piece has a well defined responsibility and they can be put together in essentially any way you’d like (of course there are a few caveats to this)
b

Bob Colner

03/26/2020, 4:02 PM
I'm a data-scientist, i guess this is a tool for a data-engineer. to much complexity outside of my core logic. I love the deam of 'negative eng.' still have a way to go to achieve that. worthy goal!
c

Chris White

03/26/2020, 4:03 PM
As a former data scientist, Prefect was designed specifically for both Data Engineers and Data Scientists 🙂; I recommend starting simple with some local development before migrating to Cloud so you can more gently get accustomed to the concepts
j

Joe Schmid

03/26/2020, 4:06 PM
@Bob Colner we use Prefect primarily for data science. Happy to answer any questions you might have -- feel free to DM me or ask on this thread.
b

Bob Colner

03/26/2020, 4:10 PM
I've got it working locally easy enough, can use
LocalDaskExecutor(scheduler = 'threads')
for parallel
I've got it working in the cloud with a single
flow.run_agent()
after that getting a full prod setup gets very eng. heavy
j

Joe Schmid

03/26/2020, 4:35 PM
@Bob Colner I think that's right, i.e. to do something that will scale really well does take some engineering, though there's a lot of community effort happening to make that easier, e.g. Dask Cloud Provider project, etc.. For us that investment was totally worthwhile and was fundamental to our strategy (train large numbers of model in parallel for R&D). If you were interested in a more turn-key solution for managed Dask take a look at Saturn Cloud as a vendor: https://www.saturncloud.io/s/ Do you already use a cloud provider like AWS, Azure, or Google Cloud Platform?
b

Bob Colner

03/26/2020, 4:43 PM
thanks, I'll take a look. I'm a one-man-band can't do it all
👍 1