https://prefect.io logo
Join Slack
Powered by
I've been using this Dockerfile for almost a year ...
# ask-community
d
I've been using this Dockerfile for almost a year now without issue, but issues have cropped up since starting to migrate to Prefect 3.
Copy code
FROM prefecthq/prefect:3-python3.12

ARG UV_EXTRA_INDEX_URL

COPY uv.lock .
COPY pyproject.toml .

COPY --from=<http://ghcr.io/astral-sh/uv:latest|ghcr.io/astral-sh/uv:latest> /uv /bin/uv

ENV UV_PROJECT_ENVIRONMENT=/opt/prefect/.venv

RUN uv sync --no-dev --frozen && uv pip install pip

ENV PATH="/opt/prefect/.venv/bin:${PATH}"
When I run a deployment using my image, it says prefect_dbt cannot be imported (doesn't exist). When I pull and enter the container, I can import fine (
python -c "import prefect_dbt"
). I can't be certain if a change in Prefect 3 deployment run steps is causing it, or a change in the Prefect 3 image. I do see use of 
uv
in the Prefect 3 image that might conflict with my 
uv
(maybe?). I've tried a few variations: remove uv image layer copy, remove venv path prepend, using /usr/bin/uv, amongst others. None have succeeded. If someone could at minimum point me to the proper image definitions for Prefect 3 vs 2 and the deployment run steps for 3 vs 2, that would be amazing.
So I got this working 
python -c "import prefect_dbt"
UV_SYSTEM_PYTHON is already set in the image
Copy code
FROM prefecthq/prefect:3-python3.12

ARG UV_EXTRA_INDEX_URL

COPY uv.lock .
COPY pyproject.toml .

RUN uv export --no-dev --frozen --all-extras > requirements.txt && uv pip install -r requirements.txt
Unfortunately, it isn't working with cloud deployments on ECS.
Copy code
Unexpected exception encountered when trying to load flow
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/opt/prefect/youtube_etl-main/src/flows/orchestrator.py", line 4, in <module>
    from prefect_dbt.cli.commands import run_dbt_build, trigger_dbt_cli_command
ModuleNotFoundError: No module named 'prefect_dbt'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/prefect/engine.py", line 112, in <module>
    flow: "Flow[..., Any]" = load_flow(flow_run)
                             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 140, in load_flow
    flow = run_coro_as_sync(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync
    return call.result()
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 329, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 402, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper
    return await task
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/client/utilities.py", line 69, in wrapper
    return await func(client, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flows.py", line 2370, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 233, in run_sync_in_worker_thread
    result = await anyio.to_thread.run_sync(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2461, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 962, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 243, in call_with_mark
    return call()
           ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flows.py", line 2059, in load_flow_from_entrypoint
    flow: Flow[P, Any] = import_object(entrypoint)  # pyright: ignore[reportRedeclaration]
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/importtools.py", line 156, in import_object
    module = load_script_as_module(script_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/importtools.py", line 121, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'src/flows/orchestrator.py' encountered an exception: ModuleNotFoundError("No module named 'prefect_dbt'")
04:50:08 PM
Error
Flow run infrastructure exited with non-zero status code:
 Exited with non 0 code. (Error Code: 1)
@Marvin Any idea what the problem could be?
m
I'm afraid I lack the context to determine what problem you're referring to. Could you provide more details about the issue you're facing, especially if it's related to Prefect or any other specific technology or task?
d
m
I'm unable to access external links, including Slack threads. However, if you can provide some details or specific questions from the thread, I'll do my best to assist you.
d
@Marvin I've been using this Dockerfile for almost a year now without issue, but issues have cropped up since starting to migrate to Prefect 3.
Copy code
FROM prefecthq/prefect:3-python3.12

ARG UV_EXTRA_INDEX_URL

COPY uv.lock .
COPY pyproject.toml .

COPY --from=<http://ghcr.io/astral-sh/uv:latest|ghcr.io/astral-sh/uv:latest> /uv /bin/uv

ENV UV_PROJECT_ENVIRONMENT=/opt/prefect/.venv

RUN uv sync --no-dev --frozen && uv pip install pip

ENV PATH="/opt/prefect/.venv/bin:${PATH}"
When I run a deployment using my image, it says prefect_dbt cannot be imported (doesn't exist). When I pull and enter the container, I can import fine (
python -c "import prefect_dbt"
). I can't be certain if a change in Prefect 3 deployment run steps is causing it, or a change in the Prefect 3 image. I do see use of 
uv
in the Prefect 3 image that might conflict with my 
uv
(maybe?). I've tried a few variations: remove uv image layer copy, remove venv path prepend, using /usr/bin/uv, amongst others. None have succeeded. If someone could at minimum point me to the proper image definitions for Prefect 3 vs 2 and the deployment run steps for 3 vs 2, that would be amazing. --- So I got this working 
python -c "import prefect_dbt"
UV_SYSTEM_PYTHON is already set in the image
Copy code
FROM prefecthq/prefect:3-python3.12

ARG UV_EXTRA_INDEX_URL

COPY uv.lock .
COPY pyproject.toml .

RUN uv export --no-dev --frozen --all-extras > requirements.txt && uv pip install -r requirements.txt
Unfortunately, it isn't working with cloud deployments on ECS.
Copy code
Unexpected exception encountered when trying to load flow
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/opt/prefect/youtube_etl-main/src/flows/orchestrator.py", line 4, in <module>
    from prefect_dbt.cli.commands import run_dbt_build, trigger_dbt_cli_command
ModuleNotFoundError: No module named 'prefect_dbt'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/prefect/engine.py", line 112, in <module>
    flow: "Flow[..., Any]" = load_flow(flow_run)
                             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flow_engine.py", line 140, in load_flow
    flow = run_coro_as_sync(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync
    return call.result()
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 329, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 402, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper
    return await task
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/client/utilities.py", line 69, in wrapper
    return await func(client, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flows.py", line 2370, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 233, in run_sync_in_worker_thread
    result = await anyio.to_thread.run_sync(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2461, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 962, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 243, in call_with_mark
    return call()
           ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/flows.py", line 2059, in load_flow_from_entrypoint
    flow: Flow[P, Any] = import_object(entrypoint)  # pyright: ignore[reportRedeclaration]
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/importtools.py", line 156, in import_object
    module = load_script_as_module(script_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/importtools.py", line 121, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'src/flows/orchestrator.py' encountered an exception: ModuleNotFoundError("No module named 'prefect_dbt'")
04:50:08 PM
Error
Flow run infrastructure exited with non-zero status code:
 Exited with non 0 code. (Error Code: 1)
m
Your message was too long by 1030 tokens...
d
God the future is bleak
d
@Dominic Tarro - any luck here?
26 Views
Open in Slack
PreviousNext
Powered by