https://prefect.io logo
a

Alvis Tang

02/04/2021, 9:54 PM
Hi a dumb question maybe. I'm trying to make my flow to run in parallel via
run_agent
but I can't make it work I tried to use
Copy code
flow.executor = LocalDaskExecutor()
flow.run()
& it can run in parallel, but not
run_agent
. any clue?
z

Zanie

02/04/2021, 9:59 PM
Hi @Alvis Tang -- I wouldn't recommend using
run_agent
it's just going to be confusing. I'd recommend starting the agent in a separate process. Your flow script should have
flow.register()
and then you'd run the flow from the UI and run an agent in a separate process to kick off the flow run. Either way, using the dask executor should be giving you the parallelism you want.
I would recommend following the tutorial at https://docs.prefect.io/orchestration/tutorial/first.html
a

Alvis Tang

02/04/2021, 10:05 PM
Thx @Zanie Yes. I think it's less confusing to just use the cli to start the agent
I'll give
prefect agent local start
a try
@Zanie It seems like that if I run
prefect agent local start
I can run 4 tasks in parallel. Is there anyway I can adjust this number? I have a HPC so it'd be great if I can run more at the same time
z

Zanie

02/04/2021, 10:48 PM
You can pass additional arguments to the executor to specify the number of threads/processes
a

Alvis Tang

02/04/2021, 11:02 PM
z

Zanie

02/04/2021, 11:03 PM
Yep the `**kwargs (Any)`: Additional keyword arguments to pass to dask config is what you'
're interested in
a

Alvis Tang

02/04/2021, 11:15 PM
thx @Zanie For the record, the argument is
LocalDaskExecutor(_num_workers_=n_workers)