Ben Epstein
02/09/2021, 10:34 PMrequirements.txt
or conda.yaml
with a job? Thanks!python_dependencies
option here but this seems like it’s for a custom use case https://docs.prefect.io/orchestration/recipes/configuring_storage.html#building-a-custom-base-imageJim Crist-Harif
02/09/2021, 10:54 PMKubernetesExecutor
, I think you're referring to a KubernetesRun
run config? This allows you to specify an image to use for a flow run. You only need to rebuild your image if any of your dependent libraries (things you import in your flow script) change. If only your flow script changes you only need to update the script wherever it's stored (e.g. github, s3, ...). If you're using docker storage though, a docker image is all you have, so that would be rebuilding an image for every change.Ben Epstein
02/09/2021, 10:55 PMJim Crist-Harif
02/09/2021, 10:56 PMIs there a way to include aWhat's the desired behavior here? Would these dependencies be installed at flow run time? That takes time, and may result in different dependencies for different runs if you aren't fully pinning the dependencies. This may be fine, but we usually encourage building an image with everything required (so you only pay the download + install cost once).orrequirements.txt
with a job?conda.yaml
prefecthq/prefect
image includes support for an EXTRA_PIP_PACKAGES
environment variable. If defined, this is executed as pip install $EXTRA_PIP_PACKAGES
before a flow run starts up. Useful for experimenting, but we'd encourage any production deployment to freeze their environment at image build time rather than installing at runtime.Ben Epstein
02/09/2021, 10:58 PMJim Crist-Harif
02/09/2021, 10:58 PMDockerRun
)
flow.run_config = DockerRun(env={"EXTRA_PIP_PACKAGES": "numpy pandas"})
Ben Epstein
02/09/2021, 11:00 PMflow.run_config = DockerRun(env={"EXTRA_PIP_PACKAGES": "numpy==1.20.1 pandas>=1.2.2"})
work?Jim Crist-Harif
02/09/2021, 11:00 PMpip install ...
works.Ben Epstein
02/09/2021, 11:00 PMJim Crist-Harif
02/09/2021, 11:00 PMBen Epstein
02/09/2021, 11:01 PM