Question about migrating to prefect cloud. How can I get multiple agent to 'listen' to my cloud flow for task parallelism? This https://docs.prefect.io/cloud/tutorial/multiple.html#run-an-agent-manually talks about agents running multiple flows but what about multiple tasks?

Hi Bob, Agents don’t run or submit tasks, they only submit Flow runs. Executors are the Prefect objects responsible for submitting tasks for execution. We have two dask-based executors for parallel task execution which can be found here: https://docs.prefect.io/api/latest/engine/executors.html For multithreading, I recommend using

LocalDaskExecutor(schduler="threads")

. This can be configured for your Flow runs in Cloud through an execution environment, as described here: https://docs.prefect.io/cloud/execution/remote_environment.html#overview

thanks, I am strugging with so many concept and the docs don't give a coherent overview.

Sounds like you would benefit from walking through our tutorial: https://docs.prefect.io/core/tutorial/01-etl-before-prefect.html

I'm using

LocalDaskExecutor(schduler="threads")

to run locally but I don't see anywhere in the docs about . how to use that with the cloud service

Check out the remote environment docs I linked to above; the

RemoteEnvironment

allows you to configure an executor for your Cloud flow runs; in particular this section can be tweaked to support the `LocalDaskExecutor`: https://docs.prefect.io/cloud/execution/remote_environment.html#examples

I've looked at the tutorials, they each let you understand a single issue / use-case but dont really let me see how all the pieces can to used together.

Prefect was designed to be like a collection of lego blocks, where each piece has a well defined responsibility and they can be put together in essentially any way you’d like (of course there are a few caveats to this)

I'm a data-scientist, i guess this is a tool for a data-engineer. to much complexity outside of my core logic. I love the deam of 'negative eng.' still have a way to go to achieve that. worthy goal!

As a former data scientist, Prefect was designed specifically for both Data Engineers and Data Scientists 🙂; I recommend starting simple with some local development before migrating to Cloud so you can more gently get accustomed to the concepts

@Bob Colner we use Prefect primarily for data science. Happy to answer any questions you might have -- feel free to DM me or ask on this thread.

I've got it working locally easy enough, can use

LocalDaskExecutor(scheduler = 'threads')

for parallel

@Bob Colner I think that's right, i.e. to do something that will scale really well does take some engineering, though there's a lot of community effort happening to make that easier, e.g. Dask Cloud Provider project, etc.. For us that investment was totally worthwhile and was fundamental to our strategy (train large numbers of model in parallel for R&D). If you were interested in a more turn-key solution for managed Dask take a look at Saturn Cloud as a vendor: https://www.saturncloud.io/s/ Do you already use a cloud provider like AWS, Azure, or Google Cloud Platform?

thanks, I'll take a look. I'm a one-man-band can't do it all