Kirill Popov
10/28/2022, 8:22 PM19:41:45.583 | INFO | prefect.agent - Submitting flow run 'd03564b9-ad4e-4310-8662-184616d6406f'
19:41:47.416 | INFO | prefect.agent - Completed submission of flow run 'd03564b9-ad4e-4310-8662-184616d6406f'
19:41:48.471 | INFO | prefect.infrastructure.kubernetes-job - Job 'expert-lyrebird-xjq42': Pod has status 'Pending'.
19:41:49.155 | INFO | prefect.infrastructure.kubernetes-job - Job 'expert-lyrebird-xjq42': Pod has status 'Running'.
19:41:52.098 | INFO | Flow run 'expert-lyrebird' - Created task run 'My Example Task-c06c9343-0' for task 'My Example Task'
19:41:52.098 | INFO | Flow run 'expert-lyrebird' - Executing 'My Example Task-c06c9343-0' immediately...
19:41:52.345 | INFO | Task run 'My Example Task-c06c9343-0' - Finished in state Completed()
19:41:52.443 | INFO | Flow run 'expert-lyrebird' - Finished in state Completed()
When running the agent locally but in docker container, the logs look similar.
Now I want the agent to run in a kubernetes deployment to automatically manage its uptime. Only 1 replica of agent pod is used.
However I only receive part of the log in the agent cli output -- it looks like no info is coming back from the worker pods back to the agent pod...
19:59:58.687 | INFO | prefect.agent - Submitting flow run 'a510b13d-b0b5-4067-8610-518e6a2b45a8'
19:59:59.268 | INFO | prefect.agent - Completed submission of flow run 'a510b13d-b0b5-4067-8610-518e6a2b45a8'
19:59:59.305 | INFO | prefect.infrastructure.kubernetes-job - Job 'secret-iguana-zlm4f': Pod has status 'Pending'.
20:00:01.349 | INFO | prefect.infrastructure.kubernetes-job - Job 'secret-iguana-zlm4f': Pod has status 'Running'.
What is going on ? how can I make the agent running on K8S produce a complete log ?
let me know if you have any ideas -- I am fairly new to k8s so might be something basic, but looks unexpected to meChristopher Boyd
10/28/2022, 8:32 PMKirill Popov
10/28/2022, 8:33 PMChristopher Boyd
10/28/2022, 8:35 PMKirill Popov
10/28/2022, 8:37 PMChristopher Boyd
10/28/2022, 8:37 PMKirill Popov
10/28/2022, 8:37 PMChristopher Boyd
10/28/2022, 8:39 PMKirill Popov
10/28/2022, 8:41 PMChristopher Boyd
10/28/2022, 8:46 PMTask run 'My Example Task-c06c9343-0' - Finished in state Completed()
19:41:52.443 | INFO | Flow run 'expert-lyrebird' - Finished in state Completed()
Kirill Popov
10/28/2022, 8:55 PMChristopher Boyd
10/28/2022, 9:21 PM21:20:27.569 | INFO | Flow run 'gifted-copperhead' - Created task run 'log_platform_info-afea9710-0' for task 'log_platform_info'
21:20:27.570 | INFO | Flow run 'gifted-copperhead' - Executing 'log_platform_info-afea9710-0' immediately...
21:20:27.711 | INFO | Task run 'log_platform_info-afea9710-0' - Host's network name = gifted-copperheadx62rw-vz42r
21:20:27.712 | INFO | Task run 'log_platform_info-afea9710-0' - Python version = 3.9.15
21:20:27.713 | INFO | Task run 'log_platform_info-afea9710-0' - Platform information (instance type) = Linux-5.4.0-1085-azure-x86_64-with-glibc2.31
21:20:27.713 | INFO | Task run 'log_platform_info-afea9710-0' - OS/Arch = linux/x86_64
21:20:27.713 | INFO | Task run 'log_platform_info-afea9710-0' - Prefect Version = 2.6.3 🚀
21:20:27.714 | INFO | Task run 'log_platform_info-afea9710-0' - Prefect API Version = 0.8.2
21:20:27.780 | INFO | Task run 'log_platform_info-afea9710-0' - Finished in state Completed()
21:20:27.900 | INFO | Flow run 'gifted-copperhead' - Finished in state Completed('All states completed.')
21:20:20.638 | INFO | prefect.agent - Completed submission of flow run 'b1988454-b944-4be4-ad4a-affa22ea1dcf'
21:20:20.745 | INFO | prefect.infrastructure.kubernetes-job - Job 'gifted-copperheadx62rw': Pod has status 'Pending'.
21:20:21.612 | INFO | prefect.infrastructure.kubernetes-job - Job 'gifted-copperheadx62rw': Pod has status 'Running'.