r

    Romain

    2 years ago
    Hi folks, I most probably miss something here, but here is an issue I encounter: while running a docker flow (similar to this one: https://docs.prefect.io/core/examples/functional_docker.html) on a Kubernetes distributed dask cluster, there is a chance that the
    GetContainerLogs
    and
    WaitOnContainer
    tasks do not execute on the same dask worker, and therefore on the same cluster node than the
    StartContainer
    task. In this case, they won't be able to access the container right? Am I missing something here? How do you handle this?
    j

    josh

    2 years ago
    Those docker tasks require the communication of a Docker daemon. I believe that if each task is given the same Docker daemon to talk to then the tasks should be able to interact with the same containers