@Marvin can you walk me through how to run a sub flow (or a deployment with run_deployment) on separate hardware? I have a flow running on a basic pod, and I have a train and eval subflow/deployment that I want to run on a k8s pod that has a GPU. I’ve tried putting the node affinity in the

prefect.yaml

file, and I’ve also tried creating a new worker pool in a new namespace with node affinity and tolerations, but so far it hasn’t worked. putting it in the

prefect.yaml

file ran the flow on the same cpu-based pod, and creating the new worker pool hasn’t successfully created a worker pod, with the error:

CreateContainerConfigError

. Can you show me how that works?

Running a subflow or a deployment on separate hardware, especially on a k8s pod with a GPU, involves several steps. Here are some general steps and tips based on the information I found: 1. Defining and Running Subflows/Deployments: You can use

run_deployment

to run a pre-existing deployment as a subflow of the calling flow. This is effectively the same as calling the flow object itself in a parent flow with something like

asyncio.gather

. 2. Deployment Files: Deployments must be defined in static files. Flows can be defined and run interactively, that is, within REPLs or Notebooks. Deployments, on the other hand, require that your flow definition be in a known file (which can be located on a remote filesystem in certain setups). 3. Prefect.yaml: With Prefect, each repo will have a single

prefect.yaml

file located at the root of the repo that contains deployment configuration for all flows in that repo. You should always run

prefect deploy

commands from the root level of your repo. 4. Node Affinity: Node affinity is a set of rules used by the scheduler to determine where a pod can be placed. The rules are defined using custom labels on nodes and label selectors specified in pods. If you're trying to run your flow on a GPU, you might need to set up node affinity in your

prefect.yaml

or in your Kubernetes configuration to ensure that your flow runs on a node with a GPU. 5. CreateContainerConfigError: This error usually indicates that there's a problem with the configuration of the container that's being created. This could be due to a variety of issues, such as problems with the image that's being used, issues with the container's environment variables, or issues with the container's command or arguments. You might need to check the logs for the pod to get more information about what's causing this error. Please note that these are general tips and the exact steps might vary based on your specific setup and requirements. If you're still having trouble, I would recommend reaching out to the Prefect community or support for more specific guidance.