Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
pacc-june-14-2023
geo-japan
pacc-london-2023
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
c
Carlo
11/06/2022, 10:28 PMHello. We just upgraded our server/agents from 2.6.4 to -> 2.6.6. We are seeing resetting connections waiting for run_deployment. Thoughts? Thread for details
✅ 1
👀 1
logs from flow run.txt
On the server side we don't see anything in the logs, just 200s from the api. We are seeing this issues pretty reliably. Running ecs fargate, for agents & flow runs. Server running on EC2. We're going to rollback to see if it helps.
b
Bianca Hoch
11/07/2022, 5:46 PMHi Carlo, thank you for raising this. I'll reach out to the team to see what could be happening here. Would you mind sharing a few example IDs of the the affected flow runs? Also, would you mind sharing a minimal reproducible example, and provide a description of how you created the deployment?
c
Carlo
11/07/2022, 7:22 PMrough_flow_def.py
We are self hosted, so I'm not sure the ids will be helpful.
We saw this for deployments that were scheduled and that we were manually kicked off from the UI.
@Bianca Hoch do you think these are related, I saw your comment in the other thread? https://github.com/PrefectHQ/prefect/issues/7424
The agent is running in an ECS task and the server is running on an EC2 instance, so two different machines. With that said, both processes are containerized.
b
Bianca Hoch
11/07/2022, 8:34 PMThank you for providing the MRE! The issue I mentioned in the other thread could potentially be related. I'll raise to the team to see if my hunch is incorrect.
c
Carlo
11/08/2022, 1:30 PMI should add when we bumped down to 2.6.4 the issue seems to have disappeared. Thanks for your help.
Hi @Bianca Hoch any update? Thanks for helping us track this down, this is currently blocking us from upgrading to the latest version.
b
Bianca Hoch
11/11/2022, 4:54 PMHi Carlo, apologies for the delay, we're still investigating on our end. Just to clarify, the only adjustment you made on your end is bumping down to 2.6.4?
c
Carlo
11/11/2022, 5:07 PMcorrect
For our server, we use the shared images, eg prefecthq/prefect:2.6.4-python3.9
We also keep our agent versions in-line with the server, but we just pip install the dependencies into our own custom image
m
Mason Menges
11/11/2022, 10:36 PMHey @Carlo we're tracking related issues for this here https://github.com/PrefectHQ/prefect/issues/7512 if you have some time to run through reporting the behavior you were seeing
🙌 1
c
Carlo
11/11/2022, 11:26 PMupdated
m
Mason Menges
11/11/2022, 11:26 PMThanks 🙏