https://prefect.io logo
Powered by
Title
a

Austin Weisgrau

02/09/2023, 7:34 PM
I want to set up a custom docker image for my flows to run in, simply based on the prefect python container with python packages pre-installed to save time at runtime. As a minimal beginning, I'm starting with my Dockerfile looking like this: 
FROM prefecthq/prefect:2-python3.10
Here is the error I get when I try and run a flow using that docker image: 
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 268, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 167, in load_flow_from_flow_run
    storage_block = Block._from_block_document(storage_document)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 571, in _from_block_document
    else cls.get_block_class_from_schema(block_document.block_schema)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 591, in get_block_class_from_schema
    return lookup_type(cls, block_schema_to_key(schema))
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/dispatch.py", line 186, in lookup_type
    raise KeyError(
KeyError: "No class found for dispatch key 's3-bucket' in registry for type 'Block'."
I'm not sure whether this is a misconfiguration of the flow deployment, the S3 block, or the Docker container/registry blocks, or something else.
✅ 1
a

Aaron Gonzalez

02/09/2023, 7:45 PM
🤔
a

Austin Weisgrau

02/09/2023, 7:45 PM
I found this error referenced in this issue, however the issue seems to be closed, and the description of the problem in the issue doesn't seem to apply to my use case. The issue describes this error with custom Block definitions, but the blocks I have defined in my deployment were all from prefect collections. I used prefect_aws.AwsCredentials, prefect_aws.s3.S3Bucket, prefect.infrastructure,docker.DockerContainer, and prefect.infrastructure,docker.DockerRegistry
My deployment code is here, I can include the block definitions too if that is helpful: 
s3_storage = S3Bucket.load("my-bucket")
s3_storage.bucket_folder = "prefect-flows"

docker_image = DockerContainer.load("my-docker-image")

helloworld_deployment = Deployment.build_from_flow(
    flow=helloworld,
    name="Hello World",
    storage=s3_storage,
    path="helloworld",
    infrastructure=docker_image,
)
a

Aaron Gonzalez

02/09/2023, 7:48 PM
Did you register those blocks?
a

Austin Weisgrau

02/09/2023, 7:51 PM
Is registration different from saving a block? The code I ran generally looked like: 
S3Bucket(
        bucket_name=bucket_name,
        aws_credentials=aws_credentials,
        bucket_folder=bucket_folder,
    ).save(bucket_name)
Generally 
Block(params).save(blockname)
a

Andrew Huang

02/09/2023, 7:54 PM
Usually this indicates the package is not installed in the infrastructure
a

Austin Weisgrau

02/09/2023, 7:54 PM
What package?
a

Andrew Huang

02/09/2023, 7:54 PM
prefect-aws is not available in the DockerContainer
a

Austin Weisgrau

02/09/2023, 7:54 PM
Ah gotcha, let me try to include that and see if that resolves it
a

Andrew Huang

02/09/2023, 7:55 PM
I believe you can bake it into your image, or use EXTRA_PIP_INSTALLS (if I remember correctly)
a

Aaron Gonzalez

02/09/2023, 7:56 PM
Ahhhh. Yeah you might just need to “RUN pip install …” in the Dockerfile and then recreate the image
a

Austin Weisgrau

02/09/2023, 7:57 PM
I did that and got a new error
11:56:22.100 | INFO    | prefect.agent - Submitting flow run '2504f6c2-4bee-43d8-a0ac-b8d5e13a2c23'
11:56:23.736 | INFO    | prefect.infrastructure.docker-container - Pulling image 'aweisgrauwfp/engineering-general-prefect'...
11:56:25.027 | INFO    | prefect.infrastructure.docker-container - Creating Docker container 'mutant-bat'...
11:56:25.048 | INFO    | prefect.infrastructure.docker-container - Docker container 'mutant-bat' has status 'created'
11:56:25.413 | INFO    | prefect.infrastructure.docker-container - Docker container 'mutant-bat' has status 'running'
11:56:25.567 | INFO    | prefect.agent - Completed submission of flow run '2504f6c2-4bee-43d8-a0ac-b8d5e13a2c23'
/usr/local/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
19:56:28.241 | INFO    | Flow run 'mutant-bat' - Downloading flow code from storage at 'helloworld'
19:56:28.824 | ERROR   | Flow run 'mutant-bat' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1016, in get_code
  File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/pipelines/helloworld/helloworld_flow.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 268, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 187, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 786, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 195, in import_object
    module = load_script_as_module(script_path)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 158, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'pipelines/helloworld/helloworld_flow.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
11:56:29.869 | INFO    | prefect.infrastructure.docker-container - Docker container 'mutant-bat' has status 'exited'
11:56:29.871 | INFO    | prefect.infrastructure.docker-container - Docker container 'mutant-bat' has status 'exited'
a

Andrew Huang

02/09/2023, 7:57 PM
Do you have an absolute path used in your script?
a

Austin Weisgrau

02/09/2023, 8:02 PM
🤔 Here are the imports where I define my deployment: 
from pipelines.helloworld.helloworld_flow import helloworld

...

helloworld_deployment = Deployment.build_from_flow(
    flow=helloworld,
    name="Hello World",
    storage=s3_storage,
    path="helloworld",
    infrastructure=docker_image,
)
a

Andrew Huang

02/09/2023, 8:02 PM
What is your flow like?
a

Austin Weisgrau

02/09/2023, 8:02 PM
import prefect

@prefect.task
def print_hello_world() -> str:
    logger = prefect.get_run_logger()
    <http://logger.info|logger.info>("Hello world.")
    return "returned_data"


@prefect.task(retries=2)
def print_second_thing(argument: str):
    logger = prefect.get_run_logger()
    <http://logger.info|logger.info>(f"Second thing: {argument}")


@prefect.flow(name="Hello World")
def helloworld() -> None:
    "First attempt at using prefect."
    data = print_hello_world()
    print_second_thing(data)


if __name__ == "__main__":
    helloworld()
a

Andrew Huang

02/09/2023, 8:03 PM
/opt/prefect/pipelines/helloworld/helloworld_flow.py
path="helloworld",
a

Austin Weisgrau

02/09/2023, 8:03 PM
Am I supposed to copy the code defining my flows into the docker image at that path?
:yess: 1
a

Andrew Huang

02/09/2023, 8:04 PM
Not sure if you need to specify path
https://docs.prefect.io/tutorials/deployments/#deployment-creation-with-python
a

Aaron Gonzalez

02/09/2023, 8:04 PM
Hey @Austin Weisgrau I did do that. Copying my flows
FROM prefecthq/prefect:2-python3.10

COPY requirements.txt .
RUN pip install --upgrade pip setuptools --no-cache-dir
RUN pip install --trusted-host <http://pypi.python.org|pypi.python.org> --no-cache-dir -r requirements.txt

ARG ENVIRONMENT
ENV ENVIRONMENT=$ENVIRONMENT

COPY flows/ /opt/prefect/flows/
a

Andrew Huang

02/09/2023, 8:04 PM
I would follow this tutorial
a

Austin Weisgrau

02/09/2023, 8:05 PM
If you have to copy your flow code into your docker container, how could anyone use the default public prefect container??
a

Andrew Huang

02/09/2023, 8:05 PM
or this tutorial: https://docs.prefect.io/tutorials/docker/?h=docker
a

Austin Weisgrau

02/09/2023, 8:05 PM
Andrew I followed the tutorials, so it's not that helpful to link them
👀 1
They don't cover how to set up a custom docker image
a

Andrew Huang

02/09/2023, 8:06 PM
Sorry to hear that; perhaps you can submit an issue to request an improvement in the docs
a

Aaron Gonzalez

02/09/2023, 8:06 PM
When you copy your local flow code into the image you and run 
docker build
you are outputting your own custom image
a

Andrew Huang

02/09/2023, 8:07 PM
You may also want to check https://discourse.prefect.io/
a

Aaron Gonzalez

02/09/2023, 8:08 PM
the default public image is a good starting point because people don't have to cobble together necessary dependencies. You start with Prefect installed and ready to go. You just need to add your workflows (which will always be custom to a user|org, no?).
a

Austin Weisgrau

02/09/2023, 8:37 PM
Ok I copied my flows into the docker image as you recommended Aaron and that seems to have resolved it
Thanks much
a

Aaron Gonzalez

02/09/2023, 8:37 PM
😛arty-parrot:
#prefect-getting-startedJoin Slack
Powered by