Emerson Franks
08/25/2022, 11:48 PMRuntimeError: File system created with scheme 'az' from base path '<az://prefect-logs>' could not be created. You are likely missing a Python module required to use the given storage protocol.
I see some mentions of adding a requirements.txt file and including adlfs in that file but that doesn't seem to get picked up using prefect deployment apply ... I also see some mentions around modifying the deployment.yaml file to include EXTRA_PIP_PACKAGES but that equally doesn't seem to do anything.
How do I get my agent to install adlfs?Christopher Boyd
08/26/2022, 7:26 PMFROM prefecthq/prefect:2.0.4-python3.9
RUN pip install adlfs
export IMAGE_REGISTRY="chaboy"
export PROJECT_NAME="prefect-azure-3.9"
export PROJECT_VERSION="latest"
echo "$IMAGE_REGISTRY/$PROJECT_NAME:$PROJECT_VERSION"
docker build --platform=linux/amd64 -t "$IMAGE_REGISTRY/$PROJECT_NAME:$PROJECT_VERSION" -f ./DOCKERFILE .
docker push "$IMAGE_REGISTRY/$PROJECT_NAME:$PROJECT_VERSION"
./docker_setup
to push the image; then you can use that image in your deployment specEmerson Franks
08/26/2022, 7:31 PMChristopher Boyd
08/26/2022, 7:33 PMEmerson Franks
08/26/2022, 11:45 PMFlow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 232, in get_filesystem_class
register_implementation(protocol, _import_class(bit["class"]))
File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 255, in _import_class
mod = importlib.import_module(mod)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'adlfs'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 328, in filesystem
self._filesystem = fsspec.filesystem(scheme, **self.settings)
File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 265, in filesystem
cls = get_filesystem_class(protocol)
File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 234, in get_filesystem_class
raise ImportError(bit["err"]) from e
ImportError: Install adlfs to access Azure Datalake Gen2 and Azure Blob Storage
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 254, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 104, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/deployments.py", line 55, in load_flow_from_flow_run
await storage_block.get_directory(from_path=None, local_path=".")
File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 574, in get_directory
return await self.filesystem.get_directory(
File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 260, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 331, in filesystem
raise RuntimeError(
RuntimeError: File system created with scheme 'az' from base path '<az://prefect-logs>' could not be created. You are likely missing a Python module required to use the given storage protocol.
Christopher Boyd
08/27/2022, 2:37 AMEmerson Franks
08/27/2022, 9:12 PM01:07:31.968 | ERROR | Flow run 'gorgeous-wallaby' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/fsspec/registry.py", line 232, in get_filesystem_class
register_implementation(protocol, _import_class(bit["class"]))
File "/usr/local/lib/python3.10/site-packages/fsspec/registry.py", line 255, in _import_class
mod = importlib.import_module(mod)
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'adlfs'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 328, in filesystem
self._filesystem = fsspec.filesystem(scheme, **self.settings)
File "/usr/local/lib/python3.10/site-packages/fsspec/registry.py", line 265, in filesystem
cls = get_filesystem_class(protocol)
File "/usr/local/lib/python3.10/site-packages/fsspec/registry.py", line 234, in get_filesystem_class
raise ImportError(bit["err"]) from e
ImportError: Install adlfs to access Azure Datalake Gen2 and Azure Blob Storage
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 254, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 104, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 55, in load_flow_from_flow_run
await storage_block.get_directory(from_path=None, local_path=".")
File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 574, in get_directory
return await self.filesystem.get_directory(
File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 260, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 331, in filesystem
raise RuntimeError(
RuntimeError: File system created with scheme 'az' from base path '<az://prefect-logs>' could not be created. You are likely missing a Python module required to use the given storage protocol.
Christopher Boyd
08/28/2022, 6:19 PM###
### A complete description of a Prefect Deployment for flow 'healthcheck'
###
name: healthy_check
description: null
version: a95236316c7c42949fcb0091ec9aaf21
# The work queue that will handle this deployment's runs
work_queue_name: default
tags:
- kubernetes
parameters: {}
schedule: null
infra_overrides: {}
infrastructure:
type: kubernetes-job
env: {}
labels: {}
name: null
command:
- python
- -m
- prefect.engine
image: chaboy/prefect-azure-3.9:latest
namespace: prefect2
service_account_name: null
image_pull_policy: null
cluster_config: null
job:
apiVersion: batch/v1
kind: Job
-i kubernetes-job
and -sb azure/newblock
Lastly, I updated the output deployment.yaml as you see above. The only two changes I made above were the namespace (prefect2, was default) and my image (chaboy/prefect-azure-3.9:latest) which is just a personal testing repoEmerson Franks
08/29/2022, 6:38 PMFlow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 254, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 104, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 55, in load_flow_from_flow_run
await storage_block.get_directory(from_path=None, local_path=".")
File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 574, in get_directory
return await self.filesystem.get_directory(
File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 260, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 111, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 96, in sync
raise return_result
File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 53, in _runner
result[0] = await coro
File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 536, in _get
rpaths = await self._expand_path(rpath, recursive=recursive)
File "/usr/local/lib/python3.10/site-packages/adlfs/spec.py", line 1532, in _expand_path
out = await self._expand_path(
File "/usr/local/lib/python3.10/site-packages/adlfs/spec.py", line 1548, in _expand_path
await self._find(p, withdirs=True, with_parent=with_parent)
File "/usr/local/lib/python3.10/site-packages/adlfs/spec.py", line 1014, in _find
infos = await self._details([b async for b in blobs])
File "/usr/local/lib/python3.10/site-packages/adlfs/spec.py", line 1014, in <listcomp>
infos = await self._details([b async for b in blobs])
File "/usr/local/lib/python3.10/site-packages/azure/core/async_paging.py", line 154, in __anext__
return await self.__anext__()
File "/usr/local/lib/python3.10/site-packages/azure/core/async_paging.py", line 157, in __anext__
self._page = await self._page_iterator.__anext__()
File "/usr/local/lib/python3.10/site-packages/azure/core/async_paging.py", line 99, in __anext__
self._response = await self._get_next(self.continuation_token)
File "/usr/local/lib/python3.10/site-packages/azure/storage/blob/aio/_list_blobs_helper.py", line 82, in _get_next_cb
process_storage_error(error)
File "/usr/local/lib/python3.10/site-packages/azure/storage/blob/_shared/response_handlers.py", line 181, in process_storage_error
exec("raise error from None") # pylint: disable=exec-used # nosec
File "<string>", line 1, in <module>
azure.core.exceptions.HttpResponseError: This request is not authorized to perform this operation.
RequestId:4525efb9-c01e-0061-7cd6-bb8f7e000000
Time:2022-08-29T18:36:17.6663274Z
ErrorCode:AuthorizationFailure
Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.
RequestId:4525efb9-c01e-0061-7cd6-bb8f7e000000
Time:2022-08-29T18:36:17.6663274Z</Message></Error>