https://prefect.io logo
Title
b

Blake Stefansen

11/15/2022, 10:00 PM
Hi Everyone, Had the same issue with k8 job deployments that was first found here
Failed to submit flow run to infrastructure
https://prefect-community.slack.com/archives/CL09KU1K7/p1663166587656539 Details in thread
:kubernetes: 1
2
k8 agent log
21:24:34.664 | INFO    | prefect.agent - Submitting flow run 
'3c1ba6ba-ed19-45f2-9d4a-144dd57aad16'
21:24:35.179 | ERROR   | prefect.agent - Failed to submit flow run 
'3c1ba6ba-ed19-45f2-9d4a-144dd57aad16' to infrastructure.
21:24:35.181 | INFO    | prefect.agent - Completed submission of flow run 
'3c1ba6ba-ed19-45f2-9d4a-144dd57aad16'
our issue was due to the length of our flow run names 2 flow run names
vAT_T_TX_TX_Broadband_Availibility_June_2022_Map_Data_RawFile2_CSV_sec6
AT_T_TX_TX_Broadband_Availibility_June_2022_Map_Data_RawFile2_CSV_6
are more than 63 characters and get cut off to
vat_t_tx_tx_broadband_availibility_june_2022_map_data_rawfile2
when creating k8 job names and k8 job labels
this causes k8 to complain and fail to create jobs (because pods with similar names already exist as far as I know) we've changed the flow run names and our issue was resolved. this took a while to debug, so I wanted to post here in case someone had a similar issue
🙌 1
🙌 2
z

Zanie

11/15/2022, 10:34 PM
The fix for this will be released on Thursday https://github.com/PrefectHQ/prefect/pull/7528
Although yours may be slightly different since the name isn’t ending in nonalphanumeric characters
b

Blake Stefansen

11/15/2022, 10:36 PM
yeah I think this is a different issue, but good thing to post 7528 here
z

Zanie

11/15/2022, 10:36 PM
It’s because the names already exist? 😕 This will be quite hard to solve if the K8s API is just throwing 500s
b

Blake Stefansen

11/15/2022, 10:42 PM
I'm not exactly sure if there is an easy fix for this, but thank you for looking anyway @Zanie! I think this an edge case. As long as my teams flow run names are less than 63 and they are all unique, I don't think we'll see this issue again