https://prefect.io logo
Join Slack
Powered by
Hello, Prefect 2.0 question: I'm trying to setup a...
# ask-community
e
Hello, Prefect 2.0 question: I'm trying to setup a kubernetes agent and I've largely been following the demo found here:

https://www.youtube.com/watch?v=m9FYoOIfWMc▾

(thanks Chris!). I think I was able to figure out the infra-block issue that Chris encountered by adding a KubernetesJob block however, I'm now getting an error message when running my flow:
Copy code
RuntimeError: File system created with scheme 'az' from base path '<az://prefect-logs>' could not be created. You are likely missing a Python module required to use the given storage protocol.
I see some mentions of adding a requirements.txt file and including adlfs in that file but that doesn't seem to get picked up using prefect deployment apply ... I also see some mentions around modifying the deployment.yaml file to include EXTRA_PIP_PACKAGES but that equally doesn't seem to do anything. How do I get my agent to install adlfs?
✅ 1
@Anna Geller / @Christopher Boyd sorry for the tags but you both always have the answers 🙂
c
Hey @Emerson Franks, I had to work around that by creating my own image
one moment
I think the engineering team is actively working on allowing EXTRA_PIP_PACKAGES to solve that in the deployment spec, but currentlyt his only works for docker containers, and not kubernetes jobs
I created my own image (granted this version is a little outdated, use the one you would like):
Copy code
FROM prefecthq/prefect:2.0.4-python3.9
RUN pip install adlfs
I create kind of a cheat file as well, to handle the registr / name / version, build and push:
Copy code
export IMAGE_REGISTRY="chaboy"
export PROJECT_NAME="prefect-azure-3.9"
export PROJECT_VERSION="latest"

echo "$IMAGE_REGISTRY/$PROJECT_NAME:$PROJECT_VERSION"
docker build --platform=linux/amd64 -t "$IMAGE_REGISTRY/$PROJECT_NAME:$PROJECT_VERSION" -f ./DOCKERFILE .
docker push "$IMAGE_REGISTRY/$PROJECT_NAME:$PROJECT_VERSION"
so I just update docker_setup file, create my dockerfile and: 
./docker_setup
to push the image; then you can use that image in your deployment spec
e
Got it this should help until it gets formally added, thanks! If your engineering team is actively working on it, I'm guessing it's not too far out?
c
I’d have to look and see where it’s currently at, I can’t say for sure how far out that is
👍 1
e
So I'm pretty sure I have things working correctly and have an image up w/ adlfs installed but I'm still getting the same error. In the screenshot, I used Lens to open a shell and get the version of adlfs that is installed (2022.7.0). As you can see from the following log though, the adlfs message is still present:
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 232, in get_filesystem_class
    register_implementation(protocol, _import_class(bit["class"]))
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 255, in _import_class
    mod = importlib.import_module(mod)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'adlfs'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 328, in filesystem
    self._filesystem = fsspec.filesystem(scheme, **self.settings)
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 265, in filesystem
    cls = get_filesystem_class(protocol)
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 234, in get_filesystem_class
    raise ImportError(bit["err"]) from e
ImportError: Install adlfs to access Azure Datalake Gen2 and Azure Blob Storage

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 254, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 104, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/deployments.py", line 55, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=None, local_path=".")
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 574, in get_directory
    return await self.filesystem.get_directory(
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 260, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 331, in filesystem
    raise RuntimeError(
RuntimeError: File system created with scheme 'az' from base path '<az://prefect-logs>' could not be created. You are likely missing a Python module required to use the given storage protocol.
and my dockerfile is ultra simple, literally what you put updated to have the same version as I have locally
the only real delta I see is that the container is running Python 3.9 but I'm not sure how that could be causing the issue.
c
Hey Emerson , sorry for a late response - late for me here. Once you built the new image, you ran a deployment build and/or updated the existing deployment.yaml to pull the new image , and applied. You're still getting the error for adlfs (and it is the new image ?) if so, we can try with matching python versions just to rule it out - if that's the case though , I can open a GitHub issue and look into it more on Monday
e
That's correct. I pulled my build, that had adlfs installed, from an ACR. I also created a new build using prefecthq/prefect:2.2.0-python3.10 as the base, thus bumping the Python version to 3.10, with the same error. I verified by using the shell to call prefect version. This is from the pod logs
Copy code
01:07:31.968 | ERROR   | Flow run 'gorgeous-wallaby' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/fsspec/registry.py", line 232, in get_filesystem_class
    register_implementation(protocol, _import_class(bit["class"]))
  File "/usr/local/lib/python3.10/site-packages/fsspec/registry.py", line 255, in _import_class
    mod = importlib.import_module(mod)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'adlfs'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 328, in filesystem
    self._filesystem = fsspec.filesystem(scheme, **self.settings)
  File "/usr/local/lib/python3.10/site-packages/fsspec/registry.py", line 265, in filesystem
    cls = get_filesystem_class(protocol)
  File "/usr/local/lib/python3.10/site-packages/fsspec/registry.py", line 234, in get_filesystem_class
    raise ImportError(bit["err"]) from e
ImportError: Install adlfs to access Azure Datalake Gen2 and Azure Blob Storage

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 254, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 104, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 55, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=None, local_path=".")
  File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 574, in get_directory
    return await self.filesystem.get_directory(
  File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 260, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
  File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 331, in filesystem
    raise RuntimeError(
RuntimeError: File system created with scheme 'az' from base path '<az://prefect-logs>' could not be created. You are likely missing a Python module required to use the given storage protocol.
😕 1
c
Very odd, I'll take a look tomorrow
👍 1
Hi @Emerson Franks, just following up, I tested this on my end, but didn’t encounter the same behavior
Here’s an example of my deployment.yaml -
Copy code
###
### A complete description of a Prefect Deployment for flow 'healthcheck'
###
name: healthy_check
description: null
version: a95236316c7c42949fcb0091ec9aaf21
# The work queue that will handle this deployment's runs
work_queue_name: default
tags:
- kubernetes
parameters: {}
schedule: null
infra_overrides: {}
infrastructure:
  type: kubernetes-job
  env: {}
  labels: {}
  name: null
  command:
  - python
  - -m
  - prefect.engine
  image: chaboy/prefect-azure-3.9:latest
  namespace: prefect2
  service_account_name: null
  image_pull_policy: null
  cluster_config: null
  job:
    apiVersion: batch/v1
    kind: Job
I did a fresh kubernetes environment (that I just deployed) and configured my api key + api url. I created a new block entirely using azure I created a fresh deployment from my flow, using 
-i kubernetes-job
and 
-sb azure/newblock
Lastly, I updated the output deployment.yaml as you see above. The only two changes I made above were the namespace (prefect2, was default) and my image (chaboy/prefect-azure-3.9:latest) which is just a personal testing repo
🙌 1
e
those arguments appear to be what I was missing, thanks! I'm still getting an error but it's at least different 🙂
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 254, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 104, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 55, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=None, local_path=".")
  File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 574, in get_directory
    return await self.filesystem.get_directory(
  File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 260, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
  File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 111, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 96, in sync
    raise return_result
  File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 536, in _get
    rpaths = await self._expand_path(rpath, recursive=recursive)
  File "/usr/local/lib/python3.10/site-packages/adlfs/spec.py", line 1532, in _expand_path
    out = await self._expand_path(
  File "/usr/local/lib/python3.10/site-packages/adlfs/spec.py", line 1548, in _expand_path
    await self._find(p, withdirs=True, with_parent=with_parent)
  File "/usr/local/lib/python3.10/site-packages/adlfs/spec.py", line 1014, in _find
    infos = await self._details([b async for b in blobs])
  File "/usr/local/lib/python3.10/site-packages/adlfs/spec.py", line 1014, in <listcomp>
    infos = await self._details([b async for b in blobs])
  File "/usr/local/lib/python3.10/site-packages/azure/core/async_paging.py", line 154, in __anext__
    return await self.__anext__()
  File "/usr/local/lib/python3.10/site-packages/azure/core/async_paging.py", line 157, in __anext__
    self._page = await self._page_iterator.__anext__()
  File "/usr/local/lib/python3.10/site-packages/azure/core/async_paging.py", line 99, in __anext__
    self._response = await self._get_next(self.continuation_token)
  File "/usr/local/lib/python3.10/site-packages/azure/storage/blob/aio/_list_blobs_helper.py", line 82, in _get_next_cb
    process_storage_error(error)
  File "/usr/local/lib/python3.10/site-packages/azure/storage/blob/_shared/response_handlers.py", line 181, in process_storage_error
    exec("raise error from None")   # pylint: disable=exec-used # nosec
  File "<string>", line 1, in <module>
azure.core.exceptions.HttpResponseError: This request is not authorized to perform this operation.
RequestId:4525efb9-c01e-0061-7cd6-bb8f7e000000
Time:2022-08-29T18:36:17.6663274Z
ErrorCode:AuthorizationFailure
Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.
RequestId:4525efb9-c01e-0061-7cd6-bb8f7e000000
Time:2022-08-29T18:36:17.6663274Z</Message></Error>
I'll have to dig into figure out if it's something wrong w/ the VPN setup or a missing creds
It was VPN related. Thanks again Chris! I really appreciate the help!
🙌 1
29 Views
Open in Slack
PreviousNext
Powered by