https://prefect.io logo
Title
r

Romain

01/30/2020, 11:36 AM
Hi folks, I most probably miss something here, but here is an issue I encounter: while running a docker flow (similar to this one: https://docs.prefect.io/core/examples/functional_docker.html) on a Kubernetes distributed dask cluster, there is a chance that the
GetContainerLogs
and
WaitOnContainer
tasks do not execute on the same dask worker, and therefore on the same cluster node than the
StartContainer
task. In this case, they won't be able to access the container right? Am I missing something here? How do you handle this?
j

josh

01/30/2020, 3:46 PM
Those docker tasks require the communication of a Docker daemon. I believe that if each task is given the same Docker daemon to talk to then the tasks should be able to interact with the same containers