Channels
#prefect-community
Title
r
Rajeev Lahoty
07/28/2023, 4:52 PMHi everyone - some of our flow runs fail with the error - Submission failed. RuntimeError: Timed out after 240.0124716758728s while watching waiting for container start. And in Azure portal, I see a container instance in Failed status and the container in waiting status in that instance. Why does it happen? How do we handle this? Are there some environmental settings we should change etc?
👀 1
c
Christopher Boyd
07/28/2023, 5:20 PMHi Rajeev,
How are these flows being deployed? Is this an agent, or a worker?
r
Rajeev Lahoty
07/28/2023, 5:21 PMHi Chris - using agent.
c
Christopher Boyd
07/28/2023, 5:24 PMHow are you deploying these flows?
r
Rajeev Lahoty
07/28/2023, 5:25 PMthe code is on github and we create deployments using python.
c
Christopher Boyd
07/28/2023, 5:25 PMcan you share the deployment code? 240s is peculiar
you likely won’t get more on the prefect side if the container fails to start as thats infrastructure provisioning
so it would be necessary to look in the resource-group where your agent is running and look at the Azure side
r
Rajeev Lahoty
07/28/2023, 5:26 PMsample deployment code
import sys, os
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(file), '../../../Prefect')))
from prefect.deployments import Deployment
from utilities.prefectconfig import get_default_settings
from flows.teams.risk_analytics_daily import team_risk_analytics_daily_process
github_block, azure_container_instance_job_block, work_pool_name, work_queue_name = get_default_settings()
deployment = Deployment.build_from_flow(
flow=team_risk_analytics_daily_process,
name = "team-risk-analytics-daily-process",
work_pool_name = work_pool_name,
work_queue_name = work_queue_name,
tags = ["teams", "loading analytics data"],
storage = github_block,
infrastructure=azure_container_instance_job_block
)
if name == "__main__":
deployment.apply()
Yeah something like 6 minutes of timeout somewhere.
Also working with infra team on our side.
c
Christopher Boyd
07/28/2023, 5:29 PMazure_container_instance_job_blockr
Rajeev Lahoty
07/28/2023, 5:30 PMI can send u screenshot if u want or if there is something specific u r looking at.
also this error is random. not on the same job or not at the same time.
c
Christopher Boyd
07/28/2023, 5:37 PMIf the error is random, i’d be suspect of Azure
👍 1
all prefect is doing is submitting an ARM template to Azure
r
Rajeev Lahoty
07/28/2023, 5:38 PMThanks Chris. Appreciate it.
c
Christopher Boyd
07/28/2023, 5:38 PMThis is for azure webapps, but I expect there is a similar config for ACI since I think they all use the same infrastructure on the Azure backend: