https://prefect.io logo
i

Isaac Brodsky

10/26/2020, 7:12 PM
Anyone familiar with flows failing because of
Unexpected error: KeyError('lz4')
? Seems the flow itself rather than a task is failing. This is using
Docker
storage, `LocalEnvironment`/`DaskExecutor` with Dask running on Kubernetes. Seems like somehow
lz4
is not present where the job is started? I do install
pyarrow
using
python_dependencies
in the
Docker
storage so I’d expect
lz4
to be there. I’m not sure where else
lz4
could be missing.
Alternately, where is
CloudFlowRunner
being run? I assume in the flow Docker image?
k

Kyle Moon-Wright

10/26/2020, 7:53 PM
Hey @Isaac Brodsky, Yes, I believe the CloudFlowRunner is run in the pulled image. Otherwise the KeyError is interesting, how did you setup your Docker storage?
Probably something like this?:
Copy code
with Flow(
    storage=Docker(
        python_dependencies=["pyarrow"]
    )
) as flow:
does it persist if we add
"lz4"
there?
👍 1
i

Isaac Brodsky

10/26/2020, 8:05 PM
I used something that looked like:
Copy code
docker = Docker(
        python_dependencies=[<elided>, 'pyarrow'],
        registry_url="<http://gcr.io/<elided>|gcr.io/<elided>>")
    my_flow.storage = docker
indeed, adding
lz4
to that list seems to fix the issue
k

Kyle Moon-Wright

10/26/2020, 8:10 PM
Huzzah! Noice. 👍
2 Views