Austin Weisgrau
02/09/2023, 7:34 PMFROM prefecthq/prefect:2-python3.10
Here is the error I get when I try and run a flow using that docker image:
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 268, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 167, in load_flow_from_flow_run
storage_block = Block._from_block_document(storage_document)
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 571, in _from_block_document
else cls.get_block_class_from_schema(block_document.block_schema)
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 591, in get_block_class_from_schema
return lookup_type(cls, block_schema_to_key(schema))
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/dispatch.py", line 186, in lookup_type
raise KeyError(
KeyError: "No class found for dispatch key 's3-bucket' in registry for type 'Block'."
I'm not sure whether this is a misconfiguration of the flow deployment, the S3 block, or the Docker container/registry blocks, or something else.Aaron Gonzalez
02/09/2023, 7:45 PMAustin Weisgrau
02/09/2023, 7:45 PMs3_storage = S3Bucket.load("my-bucket")
s3_storage.bucket_folder = "prefect-flows"
docker_image = DockerContainer.load("my-docker-image")
helloworld_deployment = Deployment.build_from_flow(
flow=helloworld,
name="Hello World",
storage=s3_storage,
path="helloworld",
infrastructure=docker_image,
)
Aaron Gonzalez
02/09/2023, 7:48 PMAustin Weisgrau
02/09/2023, 7:51 PMS3Bucket(
bucket_name=bucket_name,
aws_credentials=aws_credentials,
bucket_folder=bucket_folder,
).save(bucket_name)
Block(params).save(blockname)
Andrew Huang
02/09/2023, 7:54 PMAustin Weisgrau
02/09/2023, 7:54 PMAndrew Huang
02/09/2023, 7:54 PMAustin Weisgrau
02/09/2023, 7:54 PMAndrew Huang
02/09/2023, 7:55 PMAaron Gonzalez
02/09/2023, 7:56 PMAustin Weisgrau
02/09/2023, 7:57 PM11:56:22.100 | INFO | prefect.agent - Submitting flow run '2504f6c2-4bee-43d8-a0ac-b8d5e13a2c23'
11:56:23.736 | INFO | prefect.infrastructure.docker-container - Pulling image 'aweisgrauwfp/engineering-general-prefect'...
11:56:25.027 | INFO | prefect.infrastructure.docker-container - Creating Docker container 'mutant-bat'...
11:56:25.048 | INFO | prefect.infrastructure.docker-container - Docker container 'mutant-bat' has status 'created'
11:56:25.413 | INFO | prefect.infrastructure.docker-container - Docker container 'mutant-bat' has status 'running'
11:56:25.567 | INFO | prefect.agent - Completed submission of flow run '2504f6c2-4bee-43d8-a0ac-b8d5e13a2c23'
/usr/local/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
19:56:28.241 | INFO | Flow run 'mutant-bat' - Downloading flow code from storage at 'helloworld'
19:56:28.824 | ERROR | Flow run 'mutant-bat' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 879, in exec_module
File "<frozen importlib._bootstrap_external>", line 1016, in get_code
File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/pipelines/helloworld/helloworld_flow.py'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 268, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 187, in load_flow_from_flow_run
flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 786, in load_flow_from_entrypoint
flow = import_object(entrypoint)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 195, in import_object
module = load_script_as_module(script_path)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 158, in load_script_as_module
raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'pipelines/helloworld/helloworld_flow.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
11:56:29.869 | INFO | prefect.infrastructure.docker-container - Docker container 'mutant-bat' has status 'exited'
11:56:29.871 | INFO | prefect.infrastructure.docker-container - Docker container 'mutant-bat' has status 'exited'
Andrew Huang
02/09/2023, 7:57 PMAustin Weisgrau
02/09/2023, 8:02 PMfrom pipelines.helloworld.helloworld_flow import helloworld
...
helloworld_deployment = Deployment.build_from_flow(
flow=helloworld,
name="Hello World",
storage=s3_storage,
path="helloworld",
infrastructure=docker_image,
)
Andrew Huang
02/09/2023, 8:02 PMAustin Weisgrau
02/09/2023, 8:02 PMimport prefect
@prefect.task
def print_hello_world() -> str:
logger = prefect.get_run_logger()
<http://logger.info|logger.info>("Hello world.")
return "returned_data"
@prefect.task(retries=2)
def print_second_thing(argument: str):
logger = prefect.get_run_logger()
<http://logger.info|logger.info>(f"Second thing: {argument}")
@prefect.flow(name="Hello World")
def helloworld() -> None:
"First attempt at using prefect."
data = print_hello_world()
print_second_thing(data)
if __name__ == "__main__":
helloworld()
Andrew Huang
02/09/2023, 8:03 PM/opt/prefect/pipelines/helloworld/helloworld_flow.py
path="helloworld",
Austin Weisgrau
02/09/2023, 8:03 PMAndrew Huang
02/09/2023, 8:04 PMAaron Gonzalez
02/09/2023, 8:04 PMFROM prefecthq/prefect:2-python3.10
COPY requirements.txt .
RUN pip install --upgrade pip setuptools --no-cache-dir
RUN pip install --trusted-host <http://pypi.python.org|pypi.python.org> --no-cache-dir -r requirements.txt
ARG ENVIRONMENT
ENV ENVIRONMENT=$ENVIRONMENT
COPY flows/ /opt/prefect/flows/
Andrew Huang
02/09/2023, 8:04 PMAustin Weisgrau
02/09/2023, 8:05 PMAndrew Huang
02/09/2023, 8:05 PMAustin Weisgrau
02/09/2023, 8:05 PMAndrew Huang
02/09/2023, 8:06 PMAaron Gonzalez
02/09/2023, 8:06 PMdocker build
you are outputting your own custom imageAndrew Huang
02/09/2023, 8:07 PMAaron Gonzalez
02/09/2023, 8:08 PMAustin Weisgrau
02/09/2023, 8:37 PMAaron Gonzalez
02/09/2023, 8:37 PM