Alexis He
05/12/2022, 9:23 AMAnna Geller
05/12/2022, 11:32 AMDeploymentSpec(
name="your_deploy_name",
flow=your_flow,
tags=["local"],
flow_runner=DockerFlowRunner(
image="prefecthq/prefect:2.0b3-python3.9",
image_pull_policy="NEVER",
volumes=["/Users/anna/.aws:/root/.aws"],
),
)
here is an example I was using to interact with AWS services from a container (mounting credentials from the instance)Alexis He
05/12/2022, 11:46 AMAnna Geller
05/12/2022, 12:05 PMThe amount of data that is passed between containers is I believe not fit to be stored in Prefect storage.Storage in Prefect is for flow code, not for the actual data used in your data flows. But I can understand why this may not be clear just by looking at the name Storage Nice, I think
DockerFlowRunner
can likely fit your use case better than KubernetesFlowRunner
since it's much easier to mount an NFS volume to a Docker container than to a Kubernetes job (possible, but much harder)