Lewis Bails

08/10/2020, 8:26 AM
Hello prefectors! I'm trying to get the DaskKubernetesEnvironment to work and I'm running into what I think is a Docker storage problem. I can build the image outside of the Prefect process, so I don't think my issue is with the dockerfile.
flow.environment = DaskKubernetesEnvironment(min_workers=1s, max_workers=5) = Docker(dockerfile='./dockerfile', env_vars={'known_hosts': #known_hosts, 'ssh_prv_key': #prv_key})
flow.register(project_name='Automated training')
[2020-08-10 08:07:11] INFO - prefect.Docker | Building the flow's Docker storage...
Traceback (most recent call last):
  File "/home/leb/anaconda3/envs/ai-pipelines/lib/python3.7/site-packages/docker/api/", line 261, in _raise_for_status
  File "/home/leb/.local/lib/python3.7/site-packages/requests/", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <http+docker://localhost/v1.40/build?t=auto-dask-kube-cloud%3A2020-08-10t08-07-11-222981-00-00&q=False&nocache=False&rm=False&forcerm=True&pull=False&dockerfile=.%2Ftmpez2venih%2FDockerfile>
Can anyone help me out?

Ben Davison

08/10/2020, 8:30 AM
I've seen this error when a docker image tries to run within another docker container.

Lewis Bails

08/10/2020, 8:48 AM
Hmm thats strange. I'm definitely not running this from within a container, just your run of the mill conda environment.
I believe this is coming from the env_args parameter not working like --build-args in docker build. I needed to use the build_kwargs parameter instead, like..
build_kwargs = {'buildargs':{'known_hosts': #known_hosts}} = Docker(dockerfile='./dockerfile', build_kwargs=build_kwargs})