https://prefect.io logo
Title
b

Blake Stefansen

08/25/2022, 7:23 PM
@Anna Geller The team at my company saw this issue you opened with
s3fs
for
DockerContainer
infrastructure blocks, but this is also an issue for me with the
process
infrastructure when using prefect cloud. If I run a deployment in the cloud using
s3
for filesystem storage and
process
for infrastructure, our flow runs error out saying
ModuleNotFoundError: No module named 's3fs'
I'm not sure if this is due to our agent being a
kubernetes
agent and our deployment infrastructure being
process
, or something else. If the
process
just runs a subprocess on a docker image made by the kubernetes agent, then I would assume it's an issue with the default image used by the kubernetes agent
1
z

Zanie

08/25/2022, 7:32 PM
Hey @Blake Stefansen — you’ll just need to install
s3fs
in your agent’s environment
The default image does not include extensions specific to different cloud providers since they have large libraries
If you specify your agent command using
args
instead of
command
you can use the environment variable to install it at startup as described in that issue.
:upvote: 1
b

Blake Stefansen

08/25/2022, 8:26 PM
Thank you @Zanie I believe I have my deployment yaml setup correctly. I have the
EXTRA_PIP_PACKAGES
environment variable, and I changed
command
to
args
as suggested. However, flow runs from this deployment are simply failing with no log output. See images below
a

Anna Geller

08/26/2022, 11:11 AM
this doesn't work for KubernetesJob yet, only for DockerContainer block - you can track the status here https://github.com/PrefectHQ/prefect/issues/6551