https://prefect.io
Join Slack
Hi a dumb question maybe. I'm trying to make my flow to run in parallel via `run_agent` but I can't ...
# ask-community
a
Hi a dumb question maybe. I'm trying to make my flow to run in parallel via 
run_agent
but I can't make it work I tried to use
Copy code
flow.executor = LocalDaskExecutor()
flow.run()
& it can run in parallel, but not 
run_agent
. any clue?
z
Hi @Alvis Tang -- I wouldn't recommend using 
run_agent
it's just going to be confusing. I'd recommend starting the agent in a separate process. Your flow script should have 
flow.register()
and then you'd run the flow from the UI and run an agent in a separate process to kick off the flow run. Either way, using the dask executor should be giving you the parallelism you want.
I would recommend following the tutorial at https://docs.prefect.io/orchestration/tutorial/first.html
a
Thx @Zanie Yes. I think it's less confusing to just use the cli to start the agent
I'll give 
prefect agent local start
a try
@Zanie It seems like that if I run 
prefect agent local start
I can run 4 tasks in parallel. Is there anyway I can adjust this number? I have a HPC so it'd be great if I can run more at the same time
z
You can pass additional arguments to the executor to specify the number of threads/processes
a
z
Yep the `**kwargs (Any)`: Additional keyword arguments to pass to dask config is what you'
're interested in
a
thx @Zanie For the record, the argument is 
LocalDaskExecutor(_num_workers_=n_workers)
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by