https://prefect.io logo
#prefect-community
Title
# prefect-community
p

Poosh Adhikary

05/02/2022, 7:08 PM
Hi everyone! I am trying to execute a flow on two different docker containers with different number of CPU cores. Is it possible to dynamically set
num_workers
for LocalDaskExecutor based on the number of cores available on the container?
k

Kevin Kho

05/02/2022, 7:09 PM
Are you using a task to start the Docker container? Or are you using subflows with
DockerRun
?
p

Poosh Adhikary

05/02/2022, 7:12 PM
I am using
DockerRun
.
k

Kevin Kho

05/02/2022, 7:12 PM
Oh I get what you are saying now. You have one flow but you have two execution environments with different resources. What happens though if you leave it blank?
LocalDaskExecutor()
should infer the number of cores
p

Poosh Adhikary

05/02/2022, 7:14 PM
I do not want to use all the cores. Instead, I was hoping to do something like
num_of_cores/2
.
k

Kevin Kho

05/02/2022, 7:15 PM
You can’t with
LocalDaskExecutor
, but I think you can with
DaskExecutor
using
LocalCluster
like this
p

Poosh Adhikary

05/02/2022, 7:19 PM
Gotcha. Would have been great if we could do the same with
LocalDaskExecutor
as well. Thank you Kevin!
11 Views