Hi folks. We’re trying to get a better handle on the module and version compatibility between Flow registration -> Prefect Agent -> Flow Runner -> Dask Cluster so that we can ensure consistency without having to frequently update versions our Agent is running.
We’re trying to understand where the flow deserialization happens during a flow run on a K8s cluster. If my understanding / recall is correct
1. Agent receives the Flow and launches the Flow run pod.
2. The Flow run pod deserializes the flow from storage and launches the Dask KubeCluster.
3. The Dask scheduler pod launches worker pods based on scaling settings.
• If my understanding is correct, our Agent container needs only to have Prefect and Python versions which match the environment where we registered our Flow? As an example would an Agent running Python 3.8 and Prefect 0.14.22 work with a Flow registered using Python 3.8 and Prefect 0.15.11?
• Does the Flow run pod need to have matching dependencies for all the modules used in the environment where the flow was registered. If so, how is the image used for the Flow run pod configured? Are there dependency mismatch issues we should be aware of for Flow deserialization?
• The Dask scheduler and worker pods definitely need matching dependency versions from the registration environment to avoid any code inconsistencies.
Some of this information seems to be spread around the docs and discourse so I was hoping we could get some definitive guidance about this. Cheers