Is it possible to run mapped tasks in parallel usi...
# prefect-community
b
Is it possible to run mapped tasks in parallel using docker agent?
c
Hi Barry! Yes - the ability to parallelize tasks within a flow is independent of the type of agent you are running. This behavior (at the flow-level) is configured through an execution environment — right now to achieve parallelism in Prefect requires running your flows on Dask: https://docs.prefect.io/orchestration/execution/remote_dask_environment.html#overview
b
@Chris White thanks!
👍 1
@Chris White I have tried to use Dask but it fails with error:
Copy code
[2020-05-19 18:40:04] CRITICAL - CloudHandler | Failed to write log with error: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql/alpha (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd68468d9d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
I have a local server, a local docker agent, and local dask scheduler with two workers. It works well without dask.
c
It appears your Docker container does not have access to your Server API
b
Yes, docker run with default bridge network which has no access to Server API. Is there any option to set a network for a flow?
Or default network with which dockers with flow are deployed by agent?
c
Yup for sure; you can start docker agents with
Copy code
prefect agent start docker --network "name"
b
Hmmm… totaly have no idea why it does not have access to Server API, it is exposed on port 4200.