Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
geo-japan
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
b
Blake Stefansen
11/15/2022, 10:00 PMHi Everyone,
Had the same issue with k8 job deployments that was first found here
Failed to submit flow run to infrastructure:kubernetes: 1
✅ 2
k8 agent log
21:24:34.664 | INFO | prefect.agent - Submitting flow run
'3c1ba6ba-ed19-45f2-9d4a-144dd57aad16'
21:24:35.179 | ERROR | prefect.agent - Failed to submit flow run
'3c1ba6ba-ed19-45f2-9d4a-144dd57aad16' to infrastructure.
21:24:35.181 | INFO | prefect.agent - Completed submission of flow run
'3c1ba6ba-ed19-45f2-9d4a-144dd57aad16'our issue was due to the length of our flow run names
2 flow run names
vAT_T_TX_TX_Broadband_Availibility_June_2022_Map_Data_RawFile2_CSV_sec6AT_T_TX_TX_Broadband_Availibility_June_2022_Map_Data_RawFile2_CSV_6vat_t_tx_tx_broadband_availibility_june_2022_map_data_rawfile2this causes k8 to complain and fail to create jobs (because pods with similar names already exist as far as I know)
we've changed the flow run names and our issue was resolved.
this took a while to debug, so I wanted to post here in case someone had a similar issue
🙌 1
🙌 2
z
Zanie
11/15/2022, 10:34 PMThe fix for this will be released on Thursday https://github.com/PrefectHQ/prefect/pull/7528
Although yours may be slightly different since the name isn’t ending in nonalphanumeric characters
b
Blake Stefansen
11/15/2022, 10:36 PMyeah I think this is a different issue, but good thing to post 7528 here
z
Zanie
11/15/2022, 10:36 PMIt’s because the names already exist? 😕 This will be quite hard to solve if the K8s API is just throwing 500s
b
Blake Stefansen
11/15/2022, 10:42 PMI'm not exactly sure if there is an easy fix for this, but thank you for looking anyway @Zanie!
I think this an edge case. As long as my teams flow run names are less than 63 and they are all unique, I don't think we'll see this issue again