https://prefect.io logo
Join Slack
Powered by
Hey. I just upgraded to prefect 2.0b11 and I've an...
# prefect-community
a
Hey. I just upgraded to prefect 2.0b11 and I've an agent running in a rather restrictive kubernetes environment - meaning I can't run it as root. This results in having this warning at the Agent startup:
Copy code
/usr/local/lib/python3.9/site-packages/prefect/context.py:461: UserWarning: Failed to create the Prefect home directory at /.prefect
with SettingsContext(profile=profile, settings=new_settings) as ctx:
While Agent still starts up and picks up flow-runs, I get the following exception during flow-run
Copy code
File "/usr/local/lib/python3.9/pathlib.py", line 1323, in mkdir
self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/.prefect/storage'
I guess this has something to do with removing storage/default storage in the latest release? I configured my deployment with a RemoteFileSystem - Packager - not sure why prefect still needs to write something locally? Find my flow + deployment in the Thread.
āœ… 1
Copy code
from prefect import flow, task
from prefect.deployments import Deployment
from prefect.filesystems import RemoteFileSystem
from prefect.logging import get_run_logger
from prefect.packaging import FilePackager


@task
def greet_world():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Hello world!")


@flow
def give_greeting() -> str:
    greet_world()


minio_packager = FilePackager(filesystem=RemoteFileSystem(
    basepath="<gcs://rm-datateam-prefect-global-storage>",
    settings={
        "type":
        "service_account",
        "project_id":
        ....
    }))

Deployment(flow=give_greeting,
           name="minio_file_package_with_remote_s3fs",
           tags=["os4"],
           packager=minio_packager)

if __name__ == "__main__":
    give_greeting()
(By the way sorry for the bad variable names... I use an object called "minio_packager" for GCS facepalm )
a
Thanks for reporting that, I assume this is the place where Prefect wants to store local results now but which didn't get created automatically let me open an issue and ask the team why this folder @Marvin open "2.0b11 throws 
FileNotFoundError: [Errno 2] No such file or directory: '/.prefect/storage'
with Orion set up on Kubernetes"
m
a
Hey @Anna Geller thanks for the reply. My description was not enough, sorry. ā€¢ I use prefect 2.0 cloud ā€¢ I us an agent on kubernetes In general, if the agent has the permissions to write (eg. if the kubernetes agent is run as root), then there is not a problem. However, if I can't run the agent as root, then I oftentimes don't have these write permissions. I added these details to the github issue.
šŸ‘ 1
Just in this minute, I was able to workaround this issue, by manually creating a Dockerfile for the Agent, which manually sets permissions for this folder. Then it works. However it seems a little problematic to manually create a folder, than set permissions - already in the Agent build step.
Copy code
FROM prefecthq/prefect:2.0b11-python3.9

COPY requirements.txt .
RUN python3 -m pip install --upgrade pip && python3 -m pip install -r requirements.txt

RUN mkdir /.prefect && chgrp -R 0 /.prefect && chmod -R g=u /.prefect
šŸ‘ 1
a
I updated it too - will ask the team, thanks a lot
m
Another possibility is to add an 
emptyDir
volume and an init container in which you change permissions of the volume (example here). The manifest it applies to can be found here
šŸ™Œ 1
a
Perfect, thanks! This is a very practical solution.
šŸ™Œ 1
9 Views
Open in Slack
PreviousNext
Powered by