Anyone familiar with flows failing because of

Unexpected error: KeyError('lz4')

? Seems the flow itself rather than a task is failing. This is using

Docker

storage, `LocalEnvironment`/`DaskExecutor` with Dask running on Kubernetes. Seems like somehow

lz4

is not present where the job is started? I do install

pyarrow

using

python_dependencies

in the

Docker

storage so I’d expect

lz4

to be there. I’m not sure where else

lz4

could be missing.

Hey @Isaac Brodsky, Yes, I believe the CloudFlowRunner is run in the pulled image. Otherwise the KeyError is interesting, how did you setup your Docker storage?

I used something that looked like:

docker = Docker(
        python_dependencies=[<elided>, 'pyarrow'],
        registry_url="<http://gcr.io/<elided>|gcr.io/<elided>>")
    my_flow.storage = docker

indeed, adding

lz4

to that list seems to fix the issue

Huzzah! Noice. 👍