Isaac Brodsky
10/26/2020, 7:12 PMUnexpected error: KeyError('lz4')
? Seems the flow itself rather than a task is failing. This is using Docker
storage, `LocalEnvironment`/`DaskExecutor` with Dask running on Kubernetes. Seems like somehow lz4
is not present where the job is started? I do install pyarrow
using python_dependencies
in the Docker
storage so I’d expect lz4
to be there. I’m not sure where else lz4
could be missing.CloudFlowRunner
being run? I assume in the flow Docker image?Kyle Moon-Wright
10/26/2020, 7:53 PMwith Flow(
storage=Docker(
python_dependencies=["pyarrow"]
)
) as flow:
does it persist if we add "lz4"
there?Isaac Brodsky
10/26/2020, 8:05 PMdocker = Docker(
python_dependencies=[<elided>, 'pyarrow'],
registry_url="<http://gcr.io/<elided>|gcr.io/<elided>>")
my_flow.storage = docker
indeed, adding lz4
to that list seems to fix the issueKyle Moon-Wright
10/26/2020, 8:10 PM