Hi everyone I'm trying to get a sample flow run us...
# prefect-community
Hi everyone I'm trying to get a sample flow run using Prefect UI, S3 as storage and Kubernetes Job as Infra. Unfortunately I get the
ModuleNotFoundError: No module named 's3fs'
when running the flow. • I am not using a custom image for the agent or the the job and prefer to keep using official images. • I have specified the env for the kubernetes job in the Prefect UI but it does not make a different. • I have also tried adding the
pip install s3fs
command to the agent's deployment manifest but that doesn't make a difference either. Any ideas?
I believe this is expected. Personally I'm setting
which results in the agent running
pip install s3fs==2023.1.0
before running your job. https://docs.prefect.io/concepts/filesystems/#filesystem-package-dependencies
Thanks Timo, I managed to fix that by adding that into the deployment file:
Copy code
infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG", "EXTRA_PIP_PACKAGES":"s3fs"}},
This is indeed expected - I believe the general idea and requirement was to provide a base image from Prefect, and users could install whatever required fs package was needed for their use case, as opposed to trying to maintain multiple images with multiple packages etc. etc.
👍 1
I resolve this personally by creating a custom image that just install the fs package I need and using that