I'm having a problem with a task run using an old version of a deployment. We deployed a flow and run it. When we updated the deployment and applied it, its still running the old version. I've looked at the main-deployment.yaml and compared it to the version on S3 and its uploaded correctly. I've compared the version hash in the file to the one shown in the deployment in the UI. The flow id from the run matches the flow id of the deployment and checking the agent log the run id there matches the run id shown in the ui. I've even restarted the agent. Any ideas?

Where are you storing the flow code?

In S3. I've verified the files that get uploaded when you update the deployment are the correct uploaded files.

Ok so just to clarify, you have a flow run with deployment version 1, while that is actively running, you create a new version of the deployment (2). Are you expecting to see the active flowrun switch to using the new deployment version or are you triggering a new flow run and that is still using deployment 1

The later. The there isn't a current flow running with v1. We push v2, the deployment version id is updated, but when we run it again, it still runs v1.

hmm.. and you're triggering the flowrun from the deployment right? Not clicking the older flowrun with the retry button?

Yes. Navigate to deployments and run from there.

hmm so interesting. I'm going to test this out and see if I can reproduce it. Are you using 2.7.1?

The server and agent are 2.6.9. The user pushing the change (same one who published it originally) is >= 2.6.9. They may have updated. I'll try in a 2.7.1 environment.

Ok looking at this discourse article, that might be an issue https://discourse.prefect.io/t/does-prefect-guarantee-backwards-compatibility-between-server-orion-api-and-client-e-g-agent-process/1683

ok, I'll try either upgrading the server or downgrading the agent and pinning the versions to a specific version

😅 haa well I'm glad you got that figured out 😄