Azer Rustamov
03/16/2022, 12:36 PMAnna Geller
03/16/2022, 12:44 PMDockerFlowRunner
and didn't pass AWS credentials to your flow run container. Currently, your Docker container doesn't have the AWS credentials to pull the flow definition from S3.
2) Solution
To fix that, you can mount your AWS credentials to your DockerFlowRunner
as follows:
from prefect import flow, get_run_logger
from prefect.deployments import DeploymentSpec
from prefect.flow_runners import DockerFlowRunner
@flow
def docker_flow():
logger = get_run_logger()
<http://logger.info|logger.info>("Hello from Docker!")
DeploymentSpec(
name="example",
flow=docker_flow,
flow_runner=DockerFlowRunner(
image="prefecthq/prefect:2.0a13-python3.9",
volumes=["/Users/anna/.aws:/root/.aws"],
),
)
Note that you need to mount credentials to Docker root user, otherwise it won't work.Azer Rustamov
03/16/2022, 4:05 PMAgent started!
11:44:44.511 | INFO | prefect.agent - Submitting flow run 'ce7b588b-23ac-4757-8767-f53f412a4e6f'
11:44:44.618 | INFO | prefect.flow_runner.docker - Flow run 'celadon-cobra' has container settings = {'image': 'prefecthq/prefect:2.0a13-python3.8', 'netw
ork': None, 'command': ['python', '-m', 'prefect.engine', 'ce7b588b-23ac-4757-8767-f53f412a4e6f'], 'environment': {'PREFECT_API_URL': '<http://host.docker.int>
ernal:4200/api'}, 'auto_remove': False, 'labels': {'io.prefect.flow-run-id': 'ce7b588b-23ac-4757-8767-f53f412a4e6f'}, 'extra_hosts': {'host.docker.internal':
'host-gateway'}, 'name': 'celadon-cobra', 'volumes': []}
11:44:45.085 | INFO | prefect.agent - Completed submission of flow run 'ce7b588b-23ac-4757-8767-f53f412a4e6f'
11:44:45.095 | INFO | prefect.flow_runner.docker - Flow run container 'celadon-cobra' has status 'running'
11:44:49.756 | ERROR | prefect.engine - Engine execution of flow run 'ce7b588b-23ac-4757-8767-f53f412a4e6f' exited with unexpected exception
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 951, in <module>
enter_flow_run_engine_from_subprocess(flow_run_id)
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 134, in enter_flow_run_engine_from_subprocess
return anyio.run(retrieve_flow_then_begin_flow_run, flow_run_id)
File "/usr/local/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 56, in run
return asynclib.run(func, *args, **backend_options)
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 233, in run
return native_run(wrapper(), debug=debug)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper
return await func(*args)
File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 81, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 199, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_deployment(deployment, client=client)
File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 81, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 329, in load_flow_from_deployment
maybe_flow = await client.resolve_datadoc(deployment.flow_data)
File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 1695, in resolve_datadoc
return await resolve_inner(datadoc)
File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 1688, in resolve_inner
data = await self.retrieve_data(data)
File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 1259, in retrieve_data
return await storage_block.read(embedded_datadoc)
File "/usr/local/lib/python3.8/site-packages/prefect/blocks/storage.py", line 79, in read
return await run_sync_in_worker_thread(self._read_sync, key)
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncio.py", line 51, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(context.run, call, cancellable=True)
File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 28, in run_sync
return await get_asynclib().run_sync_in_worker_thread(func, *args, cancellable=cancellable,
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 818, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 754, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.8/site-packages/prefect/blocks/storage.py", line 89, in _read_sync
s3_client.download_fileobj(Bucket=self.bucket, Key=key, Fileobj=stream)
File "/usr/local/lib/python3.8/site-packages/boto3/s3/inject.py", line 795, in download_fileobj
return future.result()
File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 103, in result
return self._coordinator.result()
File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 266, in result
raise self._exception
File "/usr/local/lib/python3.8/site-packages/s3transfer/tasks.py", line 269, in _main
self._submit(transfer_future=transfer_future, **kwargs)
File "/usr/local/lib/python3.8/site-packages/s3transfer/download.py", line 354, in _submit
response = client.head_object(
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 395, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 725, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
11:44:50.180 | INFO | prefect.flow_runner.docker - Flow run container 'celadon-cobra' has status 'exited'
Anna Geller
03/16/2022, 4:21 PMAzer Rustamov
03/16/2022, 4:32 PM