Hello. We just upgraded our server/agents from 2.6...
# ask-communityc
Carlo
11/06/2022, 10:28 PMHello. We just upgraded our server/agents from 2.6.4 to -> 2.6.6. We are seeing resetting connections waiting for run_deployment. Thoughts? Thread for details
✅ 1
👀 1
Carlo
11/06/2022, 10:29 PMlogs from flow run.txt
Carlo
11/06/2022, 10:31 PMOn the server side we don't see anything in the logs, just 200s from the api. We are seeing this issues pretty reliably. Running ecs fargate, for agents & flow runs. Server running on EC2. We're going to rollback to see if it helps.
b
Bianca Hoch
11/07/2022, 5:46 PMHi Carlo, thank you for raising this. I'll reach out to the team to see what could be happening here. Would you mind sharing a few example IDs of the the affected flow runs? Also, would you mind sharing a minimal reproducible example, and provide a description of how you created the deployment?
c
Carlo
11/07/2022, 7:22 PMrough_flow_def.py
Carlo
11/07/2022, 7:24 PMWe are self hosted, so I'm not sure the ids will be helpful.
We saw this for deployments that were scheduled and that we were manually kicked off from the UI.
@Bianca Hoch do you think these are related, I saw your comment in the other thread? https://github.com/PrefectHQ/prefect/issues/7424
The agent is running in an ECS task and the server is running on an EC2 instance, so two different machines. With that said, both processes are containerized.
b
Bianca Hoch
11/07/2022, 8:34 PMThank you for providing the MRE! The issue I mentioned in the other thread could potentially be related. I'll raise to the team to see if my hunch is incorrect.
c
Carlo
11/08/2022, 1:30 PMI should add when we bumped down to 2.6.4 the issue seems to have disappeared. Thanks for your help.
Carlo
11/10/2022, 10:21 PMHi @Bianca Hoch any update? Thanks for helping us track this down, this is currently blocking us from upgrading to the latest version.
b
Bianca Hoch
11/11/2022, 4:54 PMHi Carlo, apologies for the delay, we're still investigating on our end. Just to clarify, the only adjustment you made on your end is bumping down to 2.6.4?
c
Carlo
11/11/2022, 5:07 PMcorrect
Carlo
11/11/2022, 5:09 PMFor our server, we use the shared images, eg prefecthq/prefect:2.6.4-python3.9
Carlo
11/11/2022, 5:10 PMWe also keep our agent versions in-line with the server, but we just pip install the dependencies into our own custom image
m
Mason Menges
11/11/2022, 10:36 PMHey @Carlo we're tracking related issues for this here https://github.com/PrefectHQ/prefect/issues/7512 if you have some time to run through reporting the behavior you were seeing
🙌 1
c
Carlo
11/11/2022, 11:26 PMupdated
m
Mason Menges
11/11/2022, 11:26 PMThanks 🙏
4 Views