https://prefect.io logo
Powered by
Title
e

eddy davies

01/25/2023, 12:41 PM
Deploying my job to kubernetes with 
prefect deployment build prefect-test/main.py:main \
    -q test \
    -n test \
    -t api-to-s3 \
    -o deployment.yaml \
    -sb s3/code \
    -ib kubernetes-job/k8s-prod
But I am getting a module not found error. I added 
"EXTRA_PIP_PACKAGES": "s3fs"
to my Kubernetes Job block environment so that it can access code on s3 but it feels wrong to add all packages in there. I have a 
pyproject.toml
and 
requirements.txt
but neither seem to be being used to install packages.
The full error code I get: 
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/prefect/beauhurst_data/generate_csv.py", line 5, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 266, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 186, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 772, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 195, in import_object
    module = load_script_as_module(script_path)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 158, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'beauhurst_data/generate_csv.py' encountered an exception: ModuleNotFoundError("No module named 'pandas'")
n

Nate

01/26/2023, 12:31 AM
Hey @eddy davies have you set the 
image
on the 
kubernetes-job/k8s-prod
to an image that has the deps in your 
requirements.txt
installed? if not, you'd need to build and push an image to a registry like ECR with some Dockerfile like 
FROM prefecthq/prefect:2.7.9-python3.{yourMinorPythonVersion}

COPY requirements.txt requirements.txt

RUN pip install -r requirements.txt
it feels wrong to add all packages in there
yep, the 
EXTRA_PIP_PACKAGES
is mostly meant for dev'ing on things and is not really ideal for a bunch of dependencies as they would install each time your flow runs, better to build a custom image that would only need to be updated when your deps change more on custom image strategies
e

eddy davies

01/26/2023, 2:14 PM
Thanks for this @Nate, I was close, using ADD instead of COPY and more importantly wasn't specifying the image version correctly. I have got an image to 
build
using 
FROM prefecthq/prefect:2.7.9-python3.10
COPY pyproject.toml pyproject.toml
RUN pip install pyproject.toml
Then 
push
that to my docker hub but trying to specify in my 
kubernetes-job
block but for some reasons it is not running, am i specifying correctly?
👍 1
j

jawnsy

01/26/2023, 2:28 PM
Try adding the tag you used (v1), so it would be: 
eddywd/prefect:v1
By default, Kubernetes will use 
latest
Also if that doesn’t work, try adding docker.io in front, like 
<http://docker.io/eddywd/prefect:v1|docker.io/eddywd/prefect:v1>
e

eddy davies

01/26/2023, 3:42 PM
All my flow runs are just getting stuck in 
Late
n

Nate

01/26/2023, 4:52 PM
what do the agent logs show?
#prefect-kubernetesJoin Slack
Powered by