Hi folks. We’re trying to get a better handle on the module and version compatibility between Flow registration -> Prefect Agent -> Flow Runner -> Dask Cluster so that we can ensure consistency without having to frequently update versions our Agent is running. We’re trying to understand where the flow deserialization happens during a flow run on a K8s cluster. If my understanding / recall is correct 1. Agent receives the Flow and launches the Flow run pod. 2. The Flow run pod deserializes the flow from storage and launches the Dask KubeCluster. 3. The Dask scheduler pod launches worker pods based on scaling settings. • If my understanding is correct, our Agent container needs only to have Prefect and Python versions which match the environment where we registered our Flow? As an example would an Agent running Python 3.8 and Prefect 0.14.22 work with a Flow registered using Python 3.8 and Prefect 0.15.11? • Does the Flow run pod need to have matching dependencies for all the modules used in the environment where the flow was registered. If so, how is the image used for the Flow run pod configured? Are there dependency mismatch issues we should be aware of for Flow deserialization? • The Dask scheduler and worker pods definitely need matching dependency versions from the registration environment to avoid any code inconsistencies. Some of this information seems to be spread around the docs and discourse so I was hoping we could get some definitive guidance about this. Cheers

For a run on Kubernetes, your written 1 or 2 or 3 are right. Agent gets the flow and spins a new pod that loads the Flow from storage. Agent versions need to be greater than or equal to the Flow storage. The Flow run pod should ideally be the same as the registration. The image can be passed to the

KubernetesRun

. There may be errors thrown if the Flow can’t be serialized properly. Dask cluster should always be matching yes.

And for clarification, the image specified in the

KubernetesRun

image

parameter is the one used in the Flow run pod?

Yes exactly! Or if you have a job template in the RunConfig, it would be the image specified there

@Kevin Kho A follow up question on Python version compatibility. Are registered flows always forward compatible with the Agent’s Python version? For instance if we registered a flow using Python 3.8 and the Agent is running 3.9 will there be an issue?

I think the agent may raise an error if the Python versions don’t match, but I can also see it working because the more important one is the version in the container of the Flow