Hello Prefect 2 0 question I m trying to setup a kubernetes

Question

Hello Prefect 2 0 question I m trying to setup a kubernetes agent and I ve largely been following the demo found here thanks Chris I think I was able to figure out the infra block issue that Chris encountered by adding a KubernetesJob block however I m now getting an error message when running my flow ```RuntimeError File system created with scheme az from base path could not be created You are likely missing a Python module required to use the given storage protocol ``` I see some mentions of adding a requirements txt file and including adlfs in that file but that doesn t seem to get picked up using prefect deployment apply I also see some mentions around modifying the deployment yaml file to include EXTRA PIP PACKAGES but that equally doesn t seem to do anything How do I get my agent to install adlfs

Emerson Franks · Answer

< Anna Geller> < Christopher Boyd> sorry for the tags but you both always have the answers slightly smiling face

Christopher Boyd · Answer

Hey < Emerson Franks> I had to work around that by creating my own image

Christopher Boyd · Answer

one moment

Christopher Boyd · Answer

I think the engineering team is actively working on allowing EXTRA PIP PACKAGES to solve that in the deployment spec but currentlyt his only works for docker containers and not kubernetes jobs

Christopher Boyd · Answer

I created my own image granted this version is a little outdated use the one you would like ```FROM prefecthq prefect 2 0 4 python3 9 RUN pip install adlfs```

Christopher Boyd · Answer

I create kind of a cheat file as well to handle the registr name version build and push ```export IMAGE REGISTRY= chaboy export PROJECT NAME= prefect azure 3 9 export PROJECT VERSION= latest echo $IMAGE REGISTRY $PROJECT NAME $PROJECT VERSION docker build platform=linux amd64 t $IMAGE REGISTRY $PROJECT NAME $PROJECT VERSION f DOCKERFILE docker push $IMAGE REGISTRY $PROJECT NAME $PROJECT VERSION ```

Christopher Boyd · Answer

so I just update docker setup file create my dockerfile and ` docker setup` to push the image then you can use that image in your deployment spec

Emerson Franks · Answer

Got it this should help until it gets formally added thanks If your engineering team is actively working on it I m guessing it s not too far out

Christopher Boyd · Answer

I d have to look and see where it s currently at I can t say for sure how far out that is

Emerson Franks · Answer

So I m pretty sure I have things working correctly and have an image up w adlfs installed but I m still getting the same error In the screenshot I used Lens to open a shell and get the version of adlfs that is installed 2022 7 0 As you can see from the following log though the adlfs message is still present ```Flow could not be retrieved from deployment Traceback most recent call last File usr local lib python3 9 site packages fsspec registry py line 232 in get filesystem class register implementation protocol import class bit class File usr local lib python3 9 site packages fsspec registry py line 255 in import class mod = importlib import module mod File usr local lib python3 9 importlib init py line 127 in import module return bootstrap gcd import name level package level File lt frozen importlib bootstrap gt line 1030 in gcd import File lt frozen importlib bootstrap gt line 1007 in find and load File lt frozen importlib bootstrap gt line 984 in find and load unlocked ModuleNotFoundError No module named adlfs The above exception was the direct cause of the following exception Traceback most recent call last File usr local lib python3 9 site packages prefect filesystems py line 328 in filesystem self filesystem = fsspec filesystem scheme self settings File usr local lib python3 9 site packages fsspec registry py line 265 in filesystem cls = get filesystem class protocol File usr local lib python3 9 site packages fsspec registry py line 234 in get filesystem class raise ImportError bit err from e ImportError Install adlfs to access Azure Datalake Gen2 and Azure Blob Storage The above exception was the direct cause of the following exception Traceback most recent call last File usr local lib python3 9 site packages prefect engine py line 254 in retrieve flow then begin flow run flow = await load flow from flow run flow run client=client File usr local lib python3 9 site packages prefect client py line 104 in with injected client return await fn args kwargs File usr local lib python3 9 site packages prefect deployments py line 55 in load flow from flow run await storage block get directory from path=None local path= File usr local lib python3 9 site packages prefect filesystems py line 574 in get directory return await self filesystem get directory File usr local lib python3 9 site packages prefect filesystems py line 260 in get directory return self filesystem get from path local path recursive=True File usr local lib python3 9 site packages prefect filesystems py line 331 in filesystem raise RuntimeError RuntimeError File system created with scheme az from base path <az prefect logs> could not be created You are likely missing a Python module required to use the given storage protocol ```

Emerson Franks · Answer

and my dockerfile is ultra simple literally what you put updated to have the same version as I have locally

Emerson Franks · Answer

the only real delta I see is that the container is running Python 3 9 but I m not sure how that could be causing the issue

Christopher Boyd · Answer

Hey Emerson sorry for a late response late for me here Once you built the new image you ran a deployment build and or updated the existing deployment yaml to pull the new image and applied You re still getting the error for adlfs and it is the new image if so we can try with matching python versions just to rule it out if that s the case though I can open a GitHub issue and look into it more on Monday

Emerson Franks · Answer

That s correct I pulled my build that had adlfs installed from an ACR I also created a new build using prefecthq prefect 2 2 0 python3 10 as the base thus bumping the Python version to 3 10 with the same error I verified by using the shell to call prefect version This is from the pod logs ```01 07 31 968 | ERROR | Flow run gorgeous wallaby Flow could not be retrieved from deployment Traceback most recent call last File usr local lib python3 10 site packages fsspec registry py line 232 in get filesystem class register implementation protocol import class bit class File usr local lib python3 10 site packages fsspec registry py line 255 in import class mod = importlib import module mod File usr local lib python3 10 importlib init py line 126 in import module return bootstrap gcd import name level package level File lt frozen importlib bootstrap gt line 1050 in gcd import File lt frozen importlib bootstrap gt line 1027 in find and load File lt frozen importlib bootstrap gt line 1004 in find and load unlocked ModuleNotFoundError No module named adlfs The above exception was the direct cause of the following exception Traceback most recent call last File usr local lib python3 10 site packages prefect filesystems py line 328 in filesystem self filesystem = fsspec filesystem scheme self settings File usr local lib python3 10 site packages fsspec registry py line 265 in filesystem cls = get filesystem class protocol File usr local lib python3 10 site packages fsspec registry py line 234 in get filesystem class raise ImportError bit err from e ImportError Install adlfs to access Azure Datalake Gen2 and Azure Blob Storage The above exception was the direct cause of the following exception Traceback most recent call last File usr local lib python3 10 site packages prefect engine py line 254 in retrieve flow then begin flow run flow = await load flow from flow run flow run client=client File usr local lib python3 10 site packages prefect client py line 104 in with injected client return await fn args kwargs File usr local lib python3 10 site packages prefect deployments py line 55 in load flow from flow run await storage block get directory from path=None local path= File usr local lib python3 10 site packages prefect filesystems py line 574 in get directory return await self filesystem get directory File usr local lib python3 10 site packages prefect filesystems py line 260 in get directory return self filesystem get from path local path recursive=True File usr local lib python3 10 site packages prefect filesystems py line 331 in filesystem raise RuntimeError RuntimeError File system created with scheme az from base path <az prefect logs> could not be created You are likely missing a Python module required to use the given storage protocol ```

Christopher Boyd · Answer

Very odd I ll take a look tomorrow

Christopher Boyd · Answer

Hi < Emerson Franks> just following up I tested this on my end but didn t encounter the same behavior

Christopher Boyd · Answer

Here s an example of my deployment yaml ``` A complete description of a Prefect Deployment for flow healthcheck name healthy check description null version a95236316c7c42949fcb0091ec9aaf21 The work queue that will handle this deployment s runs work queue name default tags kubernetes parameters schedule null infra overrides infrastructure type kubernetes job env labels name null command python m prefect engine image chaboy prefect azure 3 9 latest namespace prefect2 service account name null image pull policy null cluster config null job apiVersion batch v1 kind Job```

Christopher Boyd · Answer

I did a fresh kubernetes environment that I just deployed and configured my api key + api url I created a new block entirely using azure I created a fresh deployment from my flow using ` i kubernetes job` and ` sb azure newblock` Lastly I updated the output deployment yaml as you see above The only two changes I made above were the namespace prefect2 was default and my image chaboy prefect azure 3 9 latest which is just a personal testing repo

Emerson Franks · Answer

those arguments appear to be what I was missing thanks I m still getting an error but it s at least different slightly smiling face ```Flow could not be retrieved from deployment Traceback most recent call last File usr local lib python3 10 site packages prefect engine py line 254 in retrieve flow then begin flow run flow = await load flow from flow run flow run client=client File usr local lib python3 10 site packages prefect client py line 104 in with injected client return await fn args kwargs File usr local lib python3 10 site packages prefect deployments py line 55 in load flow from flow run await storage block get directory from path=None local path= File usr local lib python3 10 site packages prefect filesystems py line 574 in get directory return await self filesystem get directory File usr local lib python3 10 site packages prefect filesystems py line 260 in get directory return self filesystem get from path local path recursive=True File usr local lib python3 10 site packages fsspec asyn py line 111 in wrapper return sync self loop func args kwargs File usr local lib python3 10 site packages fsspec asyn py line 96 in sync raise return result File usr local lib python3 10 site packages fsspec asyn py line 53 in runner result 0 = await coro File usr local lib python3 10 site packages fsspec asyn py line 536 in get rpaths = await self expand path rpath recursive=recursive File usr local lib python3 10 site packages adlfs spec py line 1532 in expand path out = await self expand path File usr local lib python3 10 site packages adlfs spec py line 1548 in expand path await self find p withdirs=True with parent=with parent File usr local lib python3 10 site packages adlfs spec py line 1014 in find infos = await self details b async for b in blobs File usr local lib python3 10 site packages adlfs spec py line 1014 in lt listcomp gt infos = await self details b async for b in blobs File usr local lib python3 10 site packages azure core async paging py line 154 in anext return await self anext File usr local lib python3 10 site packages azure core async paging py line 157 in anext self page = await self page iterator anext File usr local lib python3 10 site packages azure core async paging py line 99 in anext self response = await self get next self continuation token File usr local lib python3 10 site packages azure storage blob aio list blobs helper py line 82 in get next cb process storage error error File usr local lib python3 10 site packages azure storage blob shared response handlers py line 181 in process storage error exec raise error from None pylint disable=exec used nosec File lt string gt line 1 in lt module gt azure core exceptions HttpResponseError This request is not authorized to perform this operation RequestId 4525efb9 c01e 0061 7cd6 bb8f7e000000 Time 2022 08 29T18 36 17 6663274Z ErrorCode AuthorizationFailure Content lt xml version= 1 0 encoding= utf 8 gt lt Error gt lt Code gt AuthorizationFailure lt Code gt lt Message gt This request is not authorized to perform this operation RequestId 4525efb9 c01e 0061 7cd6 bb8f7e000000 Time 2022 08 29T18 36 17 6663274Z lt Message gt lt Error gt ```

Emerson Franks · Answer

I ll have to dig into figure out if it s something wrong w the VPN setup or a missing creds

Emerson Franks · Answer

It was VPN related Thanks again Chris I really appreciate the help