https://prefect.io logo
Join Slack
Powered by
Hey all, I’m encountering this error when I try to...
# prefect-cloud
d
Hey all, I’m encountering this error when I try to deploy to prefect cloud:
Copy code
> Running build_docker_image step...
Step 1/10 : FROM --platform=linux/amd64 python:3.10-slim
 ---> 38ced5e623fb
Step 2/10 : WORKDIR /opt/prefect/amy-ml-server
Traceback (most recent call last):
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 154, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 125, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
    return await asyncio.wrap_future(self.future)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect_docker/deployments/steps.py", line 200, in build_docker_image
    raise BuildError(event["error"])
prefect.utilities.dockerutils.BuildError: NotFound: content digest sha256:9acd79fa542a002076583004f9dd5867a308eeff29a35fb033fdbf34cf30b58e: not found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 42, in wrapper
    return fn(*args, **kwargs)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 304, in coroutine_wrapper
    return call()
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 432, in __call__
    return self.result()
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
  File "/usr/local/Cellar/python@3.10/3.10.13_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/cli/deploy.py", line 428, in deploy
    await _run_single_deploy(
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/client/utilities.py", line 100, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/cli/deploy.py", line 657, in _run_single_deploy
    await run_steps(build_steps, step_outputs, print_function=app.console.print)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 182, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_docker.deployments.steps.build_docker_image
An exception occurred.
It used to work with the exact same prefect yaml & dockerfile, not sure why it’s stopped working. Any help would be much appreciated. Thanks!
Copy code
# Generic metadata about this project
name: amy-ml-server
prefect-version: 2.19.1

# build section allows you to manage and build docker images
build:
- prefect_docker.deployments.steps.build_docker_image:
    id: build_image
    requires: prefect-docker>=0.3.1
    image_name: <MY-ARTIFACT-DIR-HERE>
    tag: v2.2
    dockerfile: ./Dockerfile
    platform: linux/amd64

# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_docker.deployments.steps.push_docker_image:
    requires: prefect-docker>=0.3.1
    image_name: '{{ build_image.image_name }}'
    tag: '{{ build_image.tag }}'

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.set_working_directory:
    directory: /opt/prefect/amy-ml-server

# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: fdr-hal-pipeline-deploy-dev
  version:
  tags: ["dev"]
  description:
  entrypoint: flows/fdr_flow.py:process_fdr_data
  parameters:
    new_file_list: []
  work_pool:
    name: cloud-run-work-pool
    work_queue_name: default
    job_variables:
      image: '{{ build_image.image }}'
  schedules: []
^ That is my prefect.yaml file and this my dockerfile:
Copy code
# Use the official Python image as a base image
FROM --platform=linux/amd64 python:3.10-slim

# Set the working directory in the container
WORKDIR /opt/prefect/amy-ml-server

# Install system dependencies, including Git and g++
RUN apt-get update && apt-get install -y \
    gcc \
    g++ \
    libpq-dev \
    git \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Copy the requirements file into the container
COPY requirements.txt .

# Modify requirements.txt to exclude shap
RUN sed -i '/shap/d' requirements.txt

# Set Git buffer size to allow AmygdaAnalytics lib to be installed
RUN git config --global http.postBuffer 104857600

# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the working directory contents into the container
COPY . .

# Set the entrypoint for the container
ENTRYPOINT ["python"]
And this is the command I am using to deploy: 
prefect deploy --name fdr-hal-pipeline-deploy-dev --prefect-file prefect_dev.yaml
n
hi @Daniel - I believe I've seen this before, are you deploying off the same machine all the time? or is it containerized / github actions etc? if on a single machine, wondering if a
Copy code
docker system prune -a
might help
d
Hey @Nate yes I am deploying off the same machine all the time. I have tried 
docker system prune -a
and still encountering the same error
I was considering re-installing docker? Not sure if that would help as it seems to be an issue with the prefect-docker library but not 100%. Do you think a docker re-install may resolve the issue?
n
possibly, but not sure with any confidence on that personally i would want to dig into the site packages and print out the whole 
event
object here to try and debug a bit more
Copy code
File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect_docker/deployments/steps.py", line 200, in build_docker_image
    raise BuildError(event["error"])
i would suspect the issue has something to do with the state of your local docker environment somehow, but i could be wrong
d
Ok I printed the event:
Copy code
prefect deploy --name fdr-hal-pipeline-deploy-dev --prefect-file prefect_dev.yaml
Running deployment build steps...
 > Running build_docker_image step...
Step 1/10 : FROM --platform=linux/amd64 python:3.10-slim
 ---> 38ced5e623fb
Step 2/10 : WORKDIR /opt/prefect/amy-ml-server
Full event object:
{'error': 'NotFound: content digest '
          'sha256:9acd79fa542a002076583004f9dd5867a308eeff29a35fb033fdbf34cf30b58e: '
          'not found',
 'errorDetail': {'message': 'NotFound: content digest '
                            'sha256:9acd79fa542a002076583004f9dd5867a308eeff29a35fb033fdbf34cf30b58e: '
                            'not found'}}
Traceback (most recent call last):
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 154, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 125, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
    return await asyncio.wrap_future(self.future)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect_docker/deployments/steps.py", line 203, in build_docker_image
    raise BuildError(event["error"])
prefect.utilities.dockerutils.BuildError: NotFound: content digest sha256:9acd79fa542a002076583004f9dd5867a308eeff29a35fb033fdbf34cf30b58e: not found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 42, in wrapper
    return fn(*args, **kwargs)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 309, in coroutine_wrapper
    return call()
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 432, in __call__
    return self.result()
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
  File "/usr/local/Cellar/python@3.10/3.10.13_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/cli/deploy.py", line 429, in deploy
    await _run_single_deploy(
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/client/utilities.py", line 100, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/cli/deploy.py", line 658, in _run_single_deploy
    await run_steps(build_steps, step_outputs, print_function=app.console.print)
  File "/Users/dansimons/amygda/menv/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 182, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_docker.deployments.steps.build_docker_image
An exception occurred.
See lines under 
Full event object:
line @Nate
n
ok unfortunately not a lot of new info >
Copy code
NotFound: content digest sha256:9acd79fa542a002076583004f9dd5867a308eeff29a35fb033fdbf34cf30b58e: not found
if i remember correctly, this means something like "this layer of the image is inaccessible/corrupted for some reason" maybe we could try:
Copy code
docker rmi <MY-ARTIFACT-DIR-HERE>:v2.2
sorry, not sure, just guessing
d
Error response from daemon: no such image
When i run the docker rmi command
Re-installing docker seems to have resolved the issue
n
interesting! you were right - glad you got it sorted
d
Thanks for the help anyway!
n
👍
165 Views
Open in Slack
PreviousNext
Powered by