Hi everyone I'm trying to get a sample flow run using Prefect UI, S3 as storage and Kubernetes Job as Infra. Unfortunately I get the

ModuleNotFoundError: No module named 's3fs'

when running the flow. • I am not using a custom image for the agent or the the job and prefer to keep using official images. • I have specified the env for the kubernetes job in the Prefect UI but it does not make a different. • I have also tried adding the

pip install s3fs

command to the agent's deployment manifest but that doesn't make a difference either. Any ideas?

I believe this is expected. Personally I'm setting

env.EXTRA_PIP_PACKAGES

to

s3fs==2023.1.0

which results in the agent running

pip install s3fs==2023.1.0

before running your job. https://docs.prefect.io/concepts/filesystems/#filesystem-package-dependencies

Thanks Timo, I managed to fix that by adding that into the deployment file:

infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG", "EXTRA_PIP_PACKAGES":"s3fs"}},

This is indeed expected - I believe the general idea and requirement was to provide a base image from Prefect, and users could install whatever required fs package was needed for their use case, as opposed to trying to maintain multiple images with multiple packages etc. etc.