https://prefect.io
Join Slack
Hello, has anyone run a separate docker image in a flow? My prefect deployment goes as: `prefect dep...
# ask-community
s
Hello, has anyone run a separate docker image in a flow? My prefect deployment goes as: 
prefect deployment build fx_run.py:fx_rates_flow -n fx_deploy --infra docker-container -sb remote-file-system/minio
and then I also created a custom docker image with s3fs installed which I pull during deployment. However I keep getting the error: 
No such file or directory: 'docker': 'docker'
. Can anyone advise?
a
Can you use S3 instead of minio? Otherwise you may solve it by switching to process infra e.g. --infra process
The problem is docker in docker - you can google this term to read more about problems with that approach
Alternatively, you can bake your flow code into your Docker image itself - we are currently working on that feature, this way you could skip minio and have everything (storage and infra) based on the same Docker image
ETA for this is 1-2 weeks as it's a high priority request from the community
🙌 1
s
Thanks for the help. Running it as a local Process requires s3fs to be installed. Is that not required for s3 anymore and already baked into the offical image? Declaring additional EXTRA_PIP_PACKAGES in the env field, won't work as our company cluster is shielded by proxies. If I add those too, the flow isn't executed which I noticed a couple of time, i.e. when I created a separate docker agent in a Dockerfile I had to declare the http_proxy, etc first, install packages and remove the env vars again. I am not sure what is happening under the hood...
a
yup exactly, no s3fs required when flow code is baked into the image
follow the #CKNSX5WG3 - should be released rather soon
🙌 1
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by