https://prefect.io logo
Join Slack
Powered by
I want to set up a custom docker image for my flow...
# prefect-getting-started
a
I want to set up a custom docker image for my flows to run in, simply based on the prefect python container with python packages pre-installed to save time at runtime. As a minimal beginning, I'm starting with my Dockerfile looking like this:
Copy code
FROM prefecthq/prefect:2-python3.10
Here is the error I get when I try and run a flow using that docker image:
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 268, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 167, in load_flow_from_flow_run
    storage_block = Block._from_block_document(storage_document)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 571, in _from_block_document
    else cls.get_block_class_from_schema(block_document.block_schema)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 591, in get_block_class_from_schema
    return lookup_type(cls, block_schema_to_key(schema))
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/dispatch.py", line 186, in lookup_type
    raise KeyError(
KeyError: "No class found for dispatch key 's3-bucket' in registry for type 'Block'."
I'm not sure whether this is a misconfiguration of the flow deployment, the S3 block, or the Docker container/registry blocks, or something else.
✅ 1
a
🤔
a
I found this error referenced in this issue, however the issue seems to be closed, and the description of the problem in the issue doesn't seem to apply to my use case. The issue describes this error with custom Block definitions, but the blocks I have defined in my deployment were all from prefect collections. I used prefect_aws.AwsCredentials, prefect_aws.s3.S3Bucket, prefect.infrastructure,docker.DockerContainer, and prefect.infrastructure,docker.DockerRegistry
My deployment code is here, I can include the block definitions too if that is helpful:
Copy code
s3_storage = S3Bucket.load("my-bucket")
s3_storage.bucket_folder = "prefect-flows"

docker_image = DockerContainer.load("my-docker-image")

helloworld_deployment = Deployment.build_from_flow(
    flow=helloworld,
    name="Hello World",
    storage=s3_storage,
    path="helloworld",
    infrastructure=docker_image,
)
a
Did you register those blocks?
a
Is registration different from saving a block? The code I ran generally looked like:
Copy code
S3Bucket(
        bucket_name=bucket_name,
        aws_credentials=aws_credentials,
        bucket_folder=bucket_folder,
    ).save(bucket_name)
Generally 
Block(params).save(blockname)
a
Usually this indicates the package is not installed in the infrastructure
a
What package?
a
prefect-aws is not available in the DockerContainer
a
Ah gotcha, let me try to include that and see if that resolves it
a
I believe you can bake it into your image, or use EXTRA_PIP_INSTALLS (if I remember correctly)
a
Ahhhh. Yeah you might just need to “RUN pip install …” in the Dockerfile and then recreate the image
a
I did that and got a new error
Copy code
11:56:22.100 | INFO    | prefect.agent - Submitting flow run '2504f6c2-4bee-43d8-a0ac-b8d5e13a2c23'
11:56:23.736 | INFO    | prefect.infrastructure.docker-container - Pulling image 'aweisgrauwfp/engineering-general-prefect'...
11:56:25.027 | INFO    | prefect.infrastructure.docker-container - Creating Docker container 'mutant-bat'...
11:56:25.048 | INFO    | prefect.infrastructure.docker-container - Docker container 'mutant-bat' has status 'created'
11:56:25.413 | INFO    | prefect.infrastructure.docker-container - Docker container 'mutant-bat' has status 'running'
11:56:25.567 | INFO    | prefect.agent - Completed submission of flow run '2504f6c2-4bee-43d8-a0ac-b8d5e13a2c23'
/usr/local/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
19:56:28.241 | INFO    | Flow run 'mutant-bat' - Downloading flow code from storage at 'helloworld'
19:56:28.824 | ERROR   | Flow run 'mutant-bat' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1016, in get_code
  File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/pipelines/helloworld/helloworld_flow.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 268, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 187, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 786, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 195, in import_object
    module = load_script_as_module(script_path)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 158, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'pipelines/helloworld/helloworld_flow.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
11:56:29.869 | INFO    | prefect.infrastructure.docker-container - Docker container 'mutant-bat' has status 'exited'
11:56:29.871 | INFO    | prefect.infrastructure.docker-container - Docker container 'mutant-bat' has status 'exited'
a
Do you have an absolute path used in your script?
a
🤔 Here are the imports where I define my deployment:
Copy code
from pipelines.helloworld.helloworld_flow import helloworld

...

helloworld_deployment = Deployment.build_from_flow(
    flow=helloworld,
    name="Hello World",
    storage=s3_storage,
    path="helloworld",
    infrastructure=docker_image,
)
a
What is your flow like?
a
Copy code
import prefect

@prefect.task
def print_hello_world() -> str:
    logger = prefect.get_run_logger()
    <http://logger.info|logger.info>("Hello world.")
    return "returned_data"


@prefect.task(retries=2)
def print_second_thing(argument: str):
    logger = prefect.get_run_logger()
    <http://logger.info|logger.info>(f"Second thing: {argument}")


@prefect.flow(name="Hello World")
def helloworld() -> None:
    "First attempt at using prefect."
    data = print_hello_world()
    print_second_thing(data)


if __name__ == "__main__":
    helloworld()
a
/opt/prefect/pipelines/helloworld/helloworld_flow.py
Copy code
path="helloworld",
a
Am I supposed to copy the code defining my flows into the docker image at that path?
yess 1
a
Not sure if you need to specify path
a
Hey @Austin Weisgrau I did do that. Copying my flows
Copy code
FROM prefecthq/prefect:2-python3.10

COPY requirements.txt .
RUN pip install --upgrade pip setuptools --no-cache-dir
RUN pip install --trusted-host <http://pypi.python.org|pypi.python.org> --no-cache-dir -r requirements.txt

ARG ENVIRONMENT
ENV ENVIRONMENT=$ENVIRONMENT

COPY flows/ /opt/prefect/flows/
a
I would follow this tutorial
a
If you have to copy your flow code into your docker container, how could anyone use the default public prefect container??
a
a
Andrew I followed the tutorials, so it's not that helpful to link them
👀 1
They don't cover how to set up a custom docker image
a
Sorry to hear that; perhaps you can submit an issue to request an improvement in the docs
a
When you copy your local flow code into the image you and run 
docker build
you are outputting your own custom image
a
You may also want to check https://discourse.prefect.io/
a
the default public image is a good starting point because people don't have to cobble together necessary dependencies. You start with Prefect installed and ready to go. You just need to add your workflows (which will always be custom to a user|org, no?).
a
Ok I copied my flows into the docker image as you recommended Aaron and that seems to have resolved it
Thanks much
a
🦜
2 Views
Open in Slack
PreviousNext
Powered by