Hello, I am evaluating Prefect Orion for the purpose of running workflows of containers in a Kubernetes cluster; in particular, I intend to mount a volume for the containers to access (think k8s persistent volume). What are my options? • As far as I have seen, KubernetesFlowRunner does not allow me that level of customisation. • I think Prefect 1.0 allows this use case Thanks!

You're 100% right that KubernetesFlowRunner doesn't have yet all the customization that you were able to do with a KubernetesRun run config in 1.0. We are working on that - you can watch the #announcements channel in the coming months to stay up to date on that. Regarding the actual problem you try to solve, can you explain why do you need to mount some volume? What's your use case? If this is only to package code dependencies, then we will have some recipes making that easier in the near future. For Docker agent specifically, you could mount a volume on your flow runner in the DeploymentSpec as follows:

DeploymentSpec(
    name="your_deploy_name",
    flow=your_flow,
    tags=["local"],
    flow_runner=DockerFlowRunner(
        image="prefecthq/prefect:2.0b3-python3.9",
        image_pull_policy="NEVER",
        volumes=["/Users/anna/.aws:/root/.aws"],
    ),
)

here is an example I was using to interact with AWS services from a container (mounting credentials from the instance)

The use case: I expect to share a significant amount of data files between containers (in the order of GBs); coming from argo, the current scheme relies on NFS, read-write, shared volume, that is mounted through the yaml manifest that defines how containers are instantiated. The amount of data that is passed between containers is I believe not fit to be stored in Prefect storage. I will stay tuned and have a deeper look at both Prefect 1.0, and the DockerFlowRunner (in a Kubernetes environment). Thanks for your answer!

The amount of data that is passed between containers is I believe not fit to be stored in Prefect storage.

Storage in Prefect is for flow code, not for the actual data used in your data flows. But I can understand why this may not be clear just by looking at the name Storage Nice, I think

DockerFlowRunner

can likely fit your use case better than

KubernetesFlowRunner

since it's much easier to mount an NFS volume to a Docker container than to a Kubernetes job (possible, but much harder)