Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Question: I’m setting my flow to use the LocalDaskExecutor and deploying to a Docker agent. When I use a docker agent on my laptop it parallelizes just fine, but when I try to run on a VM (16 CPUs, Google Compute Engine) it’s only using a single core, and only executing one task at a time. Not sure if I’m setting it up wrong, any ideas what might be happening?

<@U02ASGFHSKC> are you using it with mapping? This should parallelize work across your threads or processes. But you can also adjust it yourself using num_workers:
```# Use 16 threads
flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=16)

# Use 16 processes
flow.executor = LocalDaskExecutor(scheduler="processes", num_workers=16)```

image.png

but using Docker agent is a bit tricky - you may need to configure Docker to use more resources, e.g. on Docker Desktop you can configure it this way:

<@U02ASGFHSKC> it looks like you can leverage the host_config to specify that on a per flow basis:
```from prefect.run_configs import DockerRun

run_config = DockerRun(host_config=dict(cpuset_cpus="0-15"))```
more info:
• <https://docs.prefect.io/api/latest/run_configs.html#dockerrun>
• <https://docker-py.readthedocs.io/en/stable/api.html#docker.api.container.ContainerApiMixin.create_host_config>

I think specifying num_workers explicitly should help here

I do have mapping, I’ll try setting the `num_workers` and see if that works better

but really, as Kevin said, try with `num_workers` first, this is likely to fix it already, without any changes on the Docker side

Screen Shot 2021-11-02 at 3.07.43 PM.png

looks like setting the `num_workers` worked. thanks!