Pedro Martins
12/22/2020, 7:24 PMaircraftlib
comes already installed but when prefect jobs start running it can't find the package.
[2020-12-22 18:10:45+0000] INFO - prefect.S3 | Downloading aircraft-etl/2020-12-22t18-10-34-141798-00-00 from dr-prefect
No module named 'aircraftlib'
Traceback (most recent call last):
File "/usr/local/bin/prefect", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/prefect/cli/execute.py", line 90, in flow_run
raise exc
File "/usr/local/lib/python3.6/site-packages/prefect/cli/execute.py", line 67, in flow_run
flow = storage.get_flow(storage.flows[flow_data.name])
File "/usr/local/lib/python3.6/site-packages/prefect/storage/s3.py", line 115, in get_flow
return cloudpickle.loads(output)
File "/usr/local/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 562, in subimport
__import__(name)
ModuleNotFoundError: No module named 'aircraftlib'
Any clue how can I make it work?Jim Crist-Harif
12/22/2020, 7:32 PMsetup.py
and pip install
it into the image?Pedro Martins
12/22/2020, 7:32 PMJim Crist-Harif
12/22/2020, 7:34 PMaircraftlib
isn't found in your image's python environment. You might try running the image locally and inspecting to see what's on your python path. Something like:
docker run -it --rm your-image /usr/bin/env bash
then start a python shell and see if you can import your module.Pedro Martins
12/22/2020, 7:37 PM$ docker run -it --rm drtools/prefect:aircraft-etl-package /usr/bin/env bash
root@058fb224350a:/# python -c "import aircraftlib; print(aircraftlib)"
<module 'aircraftlib' from '/aircraftlib/__init__.py'>
root@058fb224350a:/#
Jim Crist-Harif
12/22/2020, 7:39 PMPedro Martins
12/22/2020, 7:40 PMFROM prefecthq/prefect:0.14.0-python3.6
COPY aircraftlib ./aircraftlib
RUN conda install -e ./aircraftlib
This is my Dockerfile.
I copy to /
and pip install from there.Jim Crist-Harif
12/22/2020, 7:41 PMconda install
above be pip install
?KubernetesRun
?Pedro Martins
12/22/2020, 7:44 PMpip install
, i tried with conda but failed.custom_confs = {
"run_config": KubernetesRun(
image="drtools/prefect:aircraft-etl-package",
image_pull_secrets=["regcred"],
),
"storage": S3(bucket="dr-prefect"),
}
Jim Crist-Harif
12/22/2020, 7:45 PMprefect execute flow-run
inside it. Nothing fancy. If you could make a reproducible example, I'd be happy to take a look.Pedro Martins
12/22/2020, 7:48 PMJim Crist-Harif
12/22/2020, 7:50 PMdocker pull
errors with an auth issue.Pedro Martins
12/22/2020, 7:51 PMJim Crist-Harif
12/22/2020, 8:19 PMaircraftlib
isn't properly installed, it's only accessible from the /
directory (where the folder is). If you move the setup.py
and requirements
out a level in your module structure (so you have an aircraftlib
folder with all the code and an __init__.py
file) and a setup.py
& requirements.txt
outside this folder then things work. This is the general structure of most python packages.aircraftlib/
setup.py
requirements.txt
aircraftlib/
__init__.py
analysis.py
data
database.py
...
Pedro Martins
12/22/2020, 8:20 PMJim Crist-Harif
12/22/2020, 8:43 PMRunNamespacedJob
(https://docs.prefect.io/api/latest/tasks/kubernetes.html#runnamespacedjob), or use the raw k8s api instead.Pedro Martins
12/22/2020, 8:44 PMJim Crist-Harif
12/22/2020, 8:46 PMPedro Martins
12/22/2020, 8:46 PM