Hi Everyone,
Had the same issue with k8 job deplo...
# ask-communityb
Blake Stefansen
11/15/2022, 10:00 PMHi Everyone,
Had the same issue with k8 job deployments that was first found here
Failed to submit flow run to infrastructurekubernetes 1
✅ 2
Blake Stefansen
11/15/2022, 10:01 PMk8 agent log
Copy code
21:24:34.664 | INFO | prefect.agent - Submitting flow run
'3c1ba6ba-ed19-45f2-9d4a-144dd57aad16'
21:24:35.179 | ERROR | prefect.agent - Failed to submit flow run
'3c1ba6ba-ed19-45f2-9d4a-144dd57aad16' to infrastructure.
21:24:35.181 | INFO | prefect.agent - Completed submission of flow run
'3c1ba6ba-ed19-45f2-9d4a-144dd57aad16'Blake Stefansen
11/15/2022, 10:03 PMour issue was due to the length of our flow run names
2 flow run names
vAT_T_TX_TX_Broadband_Availibility_June_2022_Map_Data_RawFile2_CSV_sec6AT_T_TX_TX_Broadband_Availibility_June_2022_Map_Data_RawFile2_CSV_6vat_t_tx_tx_broadband_availibility_june_2022_map_data_rawfile2Blake Stefansen
11/15/2022, 10:04 PMthis causes k8 to complain and fail to create jobs (because pods with similar names already exist as far as I know)
we've changed the flow run names and our issue was resolved.
this took a while to debug, so I wanted to post here in case someone had a similar issue
🙌 1
Blake Stefansen
11/15/2022, 10:06 PM🙌 2
z
Zanie
11/15/2022, 10:34 PM
The fix for this will be released on Thursday https://github.com/PrefectHQ/prefect/pull/7528
Zanie
11/15/2022, 10:35 PM
Although yours may be slightly different since the name isn’t ending in nonalphanumeric characters
b
Blake Stefansen
11/15/2022, 10:36 PMyeah I think this is a different issue, but good thing to post 7528 here
z
Zanie
11/15/2022, 10:36 PM
It’s because the names already exist? 😕 This will be quite hard to solve if the K8s API is just throwing 500s
b
Blake Stefansen
11/15/2022, 10:42 PMI'm not exactly sure if there is an easy fix for this, but thank you for looking anyway @Zanie!
I think this an edge case. As long as my teams flow run names are less than 63 and they are all unique, I don't think we'll see this issue again
3 Views