hey folks, looking for some help here with setting up prefect 2 on K8S When running an agent in kubernetes, the agent does not receive logs from workers... among other things, this means I can not see if flow run succeeded/failed from agent logs Here is my setup: • all flows are stored in S3 • all flows run on the same infra block which is a KubernetesJob - new pod is spun up for each flow • using Prefect Cloud to orchestrate Everything was great when i was running prefect agent locally -- agent produces logs (output in CLI) similar to:

19:41:45.583 | INFO    | prefect.agent - Submitting flow run 'd03564b9-ad4e-4310-8662-184616d6406f'
19:41:47.416 | INFO    | prefect.agent - Completed submission of flow run 'd03564b9-ad4e-4310-8662-184616d6406f'
19:41:48.471 | INFO    | prefect.infrastructure.kubernetes-job - Job 'expert-lyrebird-xjq42': Pod has status 'Pending'.
19:41:49.155 | INFO    | prefect.infrastructure.kubernetes-job - Job 'expert-lyrebird-xjq42': Pod has status 'Running'.
19:41:52.098 | INFO    | Flow run 'expert-lyrebird' - Created task run 'My Example Task-c06c9343-0' for task 'My Example Task'
19:41:52.098 | INFO    | Flow run 'expert-lyrebird' - Executing 'My Example Task-c06c9343-0' immediately...
19:41:52.345 | INFO    | Task run 'My Example Task-c06c9343-0' - Finished in state Completed()
19:41:52.443 | INFO    | Flow run 'expert-lyrebird' - Finished in state Completed()

When running the agent locally but in docker container, the logs look similar. Now I want the agent to run in a kubernetes deployment to automatically manage its uptime. Only 1 replica of agent pod is used. However I only receive part of the log in the agent cli output -- it looks like no info is coming back from the worker pods back to the agent pod...

19:59:58.687 | INFO    | prefect.agent - Submitting flow run 'a510b13d-b0b5-4067-8610-518e6a2b45a8'
19:59:59.268 | INFO    | prefect.agent - Completed submission of flow run 'a510b13d-b0b5-4067-8610-518e6a2b45a8'
19:59:59.305 | INFO    | prefect.infrastructure.kubernetes-job - Job 'secret-iguana-zlm4f': Pod has status 'Pending'.
20:00:01.349 | INFO    | prefect.infrastructure.kubernetes-job - Job 'secret-iguana-zlm4f': Pod has status 'Running'.

What is going on ? how can I make the agent running on K8S produce a complete log ? let me know if you have any ideas -- I am fairly new to k8s so might be something basic, but looks unexpected to me

I think if you want logs from your jobs, you need to add logging contexts to your flow

@Christopher Boyd ok interesting…. why is there a difference between running the agent locally/in container vs in k8s deployment then?

They would be using different infrastructure for execution, locally it’s using the process, docker I’d have to check - I guess more specifically what logs do you want to come from your agent log

@Christopher Boyd the execution of the flow is always on KubernetesJob infra block in all 3 cases, only agent execution environment changes

So you want the flow logs ?

log-wise i’m only looking for status changes of flows & tasks similar to the first quote streamed back to the agent (at least now)

@Christopher Boyd i am using Prefect Cloud for job metadata/orchestration, kubernetes in digitalocean cloud for flow execution & now want to use the same kubernetes for hosting the agent

The info that you’re missing from the pods / flows, are being emitted from the flows / tasks themselves:

Task run 'My Example Task-c06c9343-0' - Finished in state Completed()
19:41:52.443 | INFO    | Flow run 'expert-lyrebird' - Finished in state Completed()

thanks @Christopher Boyd still confused as to why is there a difference where do I run the agent, will try to see if adding explicit logging changes anything

@Kirill Popov - when I add the logging to my flow, this is what I get out of the actual pod / job:

21:20:27.569 | INFO    | Flow run 'gifted-copperhead' - Created task run 'log_platform_info-afea9710-0' for task 'log_platform_info'
21:20:27.570 | INFO    | Flow run 'gifted-copperhead' - Executing 'log_platform_info-afea9710-0' immediately...
21:20:27.711 | INFO    | Task run 'log_platform_info-afea9710-0' - Host's network name = gifted-copperheadx62rw-vz42r
21:20:27.712 | INFO    | Task run 'log_platform_info-afea9710-0' - Python version = 3.9.15
21:20:27.713 | INFO    | Task run 'log_platform_info-afea9710-0' - Platform information (instance type) = Linux-5.4.0-1085-azure-x86_64-with-glibc2.31 
21:20:27.713 | INFO    | Task run 'log_platform_info-afea9710-0' - OS/Arch = linux/x86_64
21:20:27.713 | INFO    | Task run 'log_platform_info-afea9710-0' - Prefect Version = 2.6.3 🚀
21:20:27.714 | INFO    | Task run 'log_platform_info-afea9710-0' - Prefect API Version = 0.8.2
21:20:27.780 | INFO    | Task run 'log_platform_info-afea9710-0' - Finished in state Completed()
21:20:27.900 | INFO    | Flow run 'gifted-copperhead' - Finished in state Completed('All states completed.')