Abhinav Chordia
04/18/2023, 3:53 PMDeployment.build_from_flow(
flow=flow,
name=name,
work_queue_name=work_queue,
path=f"/user/{os.getlogin()}/flow/{flow.__name__}/deployments/{name}",
infrastructure=get_infra_block(infra_block),
storage=get_storage_block(storage_block),
infra_overrides={
"env": {
"PREFECT_LOGGING_LEVEL": "DEBUG",
# "EXTRA_PIP_PACKAGES":"adlfs"
},
"image" : "<http://docker.gh.st/hypo:dev|docker.gh.st/hypo:dev>"
},
)
where the image is built like this:
# syntax=docker/dockerfile:1
FROM --platform=linux/amd64 prefecthq/prefect:2-python3.9-kubernetes
COPY pyproject.toml .
RUN pip install --no-cache-dir --upgrade pip && pip install pip-tools && python -m piptools compile --resolver=backtracking pyproject.toml && pip install -r requirements.txt
When I try to run the flow, I get the following error:
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 846, in exec_module
File "<frozen importlib._bootstrap_external>", line 982, in get_code
File "<frozen importlib._bootstrap_external>", line 1039, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'src/hypo/flows/example/example_flow.py'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 276, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.9/site-packages/prefect/client/utilities.py", line 40, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/deployments.py", line 217, in load_flow_from_flow_run
flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.9/site-packages/prefect/flows.py", line 809, in load_flow_from_entrypoint
flow = import_object(entrypoint)
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 201, in import_object
module = load_script_as_module(script_path)
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 164, in load_script_as_module
raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'src/hypo/flows/example/example_flow.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
Deployment.build_from_flow(
flow=flow,
name=name,
work_queue_name=work_queue,
path=f"/user/{os.getlogin()}/flow/{flow.__name__}/deployments/{name}",
infrastructure=get_infra_block(infra_block),
storage=get_storage_block(storage_block),
infra_overrides={
"env": {
"PREFECT_LOGGING_LEVEL": "DEBUG",
"EXTRA_PIP_PACKAGES":"adlfs"
},
},
)
Where I’m just using the default prefect image.Ryan Peden
04/18/2023, 8:15 PMEXTRA_PIP_PACKAGES
before running the deployment.
Here's the Dockerfile Prefect uses for this. You'll find the entrypoint right down at the bottom.
Perhaps using this script or something similar would help.
I see you're already installing dependencies from a requirements.txt, though. If adlfs
is the only thing missing, you could also just install it when building your container image so you won't need to use EXTRA_PIP_PACKAGES
.Abhinav Chordia
04/18/2023, 9:31 PMRyan Peden
04/18/2023, 9:34 PMAbhinav Chordia
04/18/2023, 9:36 PM