Thread
#prefect-community
    b

    Barry Roszak

    2 years ago
    Is it possible to run mapped tasks in parallel using docker agent?
    Chris White

    Chris White

    2 years ago
    Hi Barry! Yes - the ability to parallelize tasks within a flow is independent of the type of agent you are running. This behavior (at the flow-level) is configured through an execution environment — right now to achieve parallelism in Prefect requires running your flows on Dask: https://docs.prefect.io/orchestration/execution/remote_dask_environment.html#overview
    b

    Barry Roszak

    2 years ago
    @Chris White thanks!
    @Chris White I have tried to use Dask but it fails with error:
    [2020-05-19 18:40:04] CRITICAL - CloudHandler | Failed to write log with error: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql/alpha (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd68468d9d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
    I have a local server, a local docker agent, and local dask scheduler with two workers. It works well without dask.
    Chris White

    Chris White

    2 years ago
    It appears your Docker container does not have access to your Server API
    b

    Barry Roszak

    2 years ago
    Yes, docker run with default bridge network which has no access to Server API. Is there any option to set a network for a flow?
    Or default network with which dockers with flow are deployed by agent?
    Chris White

    Chris White

    2 years ago
    Yup for sure; you can start docker agents with
    prefect agent start docker --network "name"
    b

    Barry Roszak

    2 years ago
    Hmmm… totaly have no idea why it does not have access to Server API, it is exposed on port 4200.