Hello prefectors!
I'm trying to get the DaskKuber...
# ask-communityl
Lewis Bails
08/10/2020, 8:26 AMHello prefectors!
I'm trying to get the DaskKubernetesEnvironment to work and I'm running into what I think is a Docker storage problem.
I can build the image outside of the Prefect process, so I don't think my issue is with the dockerfile.
Copy code
flow.environment = DaskKubernetesEnvironment(min_workers=1s, max_workers=5)
flow.storage = Docker(dockerfile='./dockerfile', env_vars={'known_hosts': #known_hosts, 'ssh_prv_key': #prv_key})
flow.register(project_name='Automated training') Copy code
[2020-08-10 08:07:11] INFO - prefect.Docker | Building the flow's Docker storage...
Traceback (most recent call last):
File "/home/leb/anaconda3/envs/ai-pipelines/lib/python3.7/site-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/home/leb/.local/lib/python3.7/site-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <http+docker://localhost/v1.40/build?t=auto-dask-kube-cloud%3A2020-08-10t08-07-11-222981-00-00&q=False&nocache=False&rm=False&forcerm=True&pull=False&dockerfile=.%2Ftmpez2venih%2FDockerfile>b
Ben Davison
08/10/2020, 8:30 AMI've seen this error when a docker image tries to run within another docker container.
l
Lewis Bails
08/10/2020, 8:48 AMHmm thats strange. I'm definitely not running this from within a container, just your run of the mill conda environment.
Lewis Bails
08/10/2020, 9:15 AMI believe this is coming from the env_args parameter not working like --build-args in docker build.
I needed to use the build_kwargs parameter instead, like..
Copy code
build_kwargs = {'buildargs':{'known_hosts': #known_hosts}}
flow.storage = Docker(dockerfile='./dockerfile', build_kwargs=build_kwargs})