eddy davies
01/25/2023, 12:41 PMprefect deployment build prefect-test/main.py:main \
-q test \
-n test \
-t api-to-s3 \
-o deployment.yaml \
-sb s3/code \
-ib kubernetes-job/k8s-prod
But I am getting a module not found error. I added "EXTRA_PIP_PACKAGES": "s3fs"
to my Kubernetes Job block environment so that it can access code on s3 but it feels wrong to add all packages in there. I have a pyproject.toml
and requirements.txt
but neither seem to be being used to install packages.Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/opt/prefect/beauhurst_data/generate_csv.py", line 5, in <module>
import pandas as pd
ModuleNotFoundError: No module named 'pandas'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 266, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 186, in load_flow_from_flow_run
flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 772, in load_flow_from_entrypoint
flow = import_object(entrypoint)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 195, in import_object
module = load_script_as_module(script_path)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 158, in load_script_as_module
raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'beauhurst_data/generate_csv.py' encountered an exception: ModuleNotFoundError("No module named 'pandas'")
Nate
01/26/2023, 12:31 AMimage
on the kubernetes-job/k8s-prod
to an image that has the deps in your requirements.txt
installed?
if not, you'd need to build and push an image to a registry like ECR with some Dockerfile like
FROM prefecthq/prefect:2.7.9-python3.{yourMinorPythonVersion}
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
it feels wrong to add all packages in thereyep, the
EXTRA_PIP_PACKAGES
is mostly meant for dev'ing on things and is not really ideal for a bunch of dependencies as they would install each time your flow runs, better to build a custom image that would only need to be updated when your deps change
more on custom image strategieseddy davies
01/26/2023, 2:14 PMbuild
using
FROM prefecthq/prefect:2.7.9-python3.10
COPY pyproject.toml pyproject.toml
RUN pip install pyproject.toml
push
that to my docker hub but trying to specify in my kubernetes-job
block but for some reasons it is not running, am i specifying correctly?jawnsy
01/26/2023, 2:28 PMeddywd/prefect:v1
By default, Kubernetes will use latest
Also if that doesn’t work, try adding docker.io in front, like <http://docker.io/eddywd/prefect:v1|docker.io/eddywd/prefect:v1>
eddy davies
01/26/2023, 3:42 PMLate
Nate
01/26/2023, 4:52 PM