Is it possible to run mapped tasks in parallel using docker

Question

Is it possible to run mapped tasks in parallel using docker agent

Chris White · Answer

Hi Barry Yes the ability to parallelize tasks within a flow is independent of the type of agent you are running This behavior at the flow level is configured through an execution environment right now to achieve parallelism in Prefect requires running your flows on Dask <https docs prefect io orchestration execution remote dask environment html overview>

Barry Roszak · Answer

< Chris White> thanks

Barry Roszak · Answer

< Chris White> I have tried to use Dask but it fails with error ``` 2020 05 19 18 40 04 CRITICAL CloudHandler | Failed to write log with error HTTPConnectionPool host= host docker internal port=4200 Max retries exceeded with url graphql alpha Caused by NewConnectionError lt urllib3 connection HTTPConnection object at 0x7fd68468d9d0 gt Failed to establish a new connection Errno 2 Name or service not known ``` I have a local server a local docker agent and local dask scheduler with two workers It works well without dask

Chris White · Answer

It appears your Docker container does not have access to your Server API

Barry Roszak · Answer

Yes docker run with default bridge network which has no access to Server API Is there any option to set a network for a flow

Barry Roszak · Answer

Or default network with which dockers with flow are deployed by agent

Chris White · Answer

Yup for sure you can start docker agents with ```prefect agent start docker network name ```

Barry Roszak · Answer

Hmmm totaly have no idea why it does not have access to Server API it is exposed on port 4200