hey folks looking for some help here with setting up prefect

Question

hey folks looking for some help here with setting up prefect 2 on K8S When running an agent in kubernetes the agent does not receive logs from workers among other things this means I can not see if flow run succeeded failed from agent logs Here is my setup all flows are stored in S3 all flows run on the same infra block which is a KubernetesJob new pod is spun up for each flow using Prefect Cloud to orchestrate Everything was great when i was running prefect agent locally agent produces logs output in CLI similar to ```19 41 45 583 | INFO | prefect agent Submitting flow run d03564b9 ad4e 4310 8662 184616d6406f 19 41 47 416 | INFO | prefect agent Completed submission of flow run d03564b9 ad4e 4310 8662 184616d6406f 19 41 48 471 | INFO | prefect infrastructure kubernetes job Job expert lyrebird xjq42 Pod has status Pending 19 41 49 155 | INFO | prefect infrastructure kubernetes job Job expert lyrebird xjq42 Pod has status Running 19 41 52 098 | INFO | Flow run expert lyrebird Created task run My Example Task c06c9343 0 for task My Example Task 19 41 52 098 | INFO | Flow run expert lyrebird Executing My Example Task c06c9343 0 immediately 19 41 52 345 | INFO | Task run My Example Task c06c9343 0 Finished in state Completed 19 41 52 443 | INFO | Flow run expert lyrebird Finished in state Completed ``` When running the agent locally but in docker container the logs look similar Now I want the agent to run in a kubernetes deployment to automatically manage its uptime Only 1 replica of agent pod is used However I only receive part of the log in the agent cli output it looks like no info is coming back from the worker pods back to the agent pod ```19 59 58 687 | INFO | prefect agent Submitting flow run a510b13d b0b5 4067 8610 518e6a2b45a8 19 59 59 268 | INFO | prefect agent Completed submission of flow run a510b13d b0b5 4067 8610 518e6a2b45a8 19 59 59 305 | INFO | prefect infrastructure kubernetes job Job secret iguana zlm4f Pod has status Pending 20 00 01 349 | INFO | prefect infrastructure kubernetes job Job secret iguana zlm4f Pod has status Running ``` What is going on how can I make the agent running on K8S produce a complete log let me know if you have any ideas I am fairly new to k8s so might be something basic but looks unexpected to me

Christopher Boyd · Answer

I think if you want logs from your jobs you need to add logging contexts to your flow

Christopher Boyd · Answer

This is expected behavior

Christopher Boyd · Answer

You need a task logger flow logger in the flow to emit logs from the jobs pods

Kirill Popov · Answer

< Christopher Boyd> ok interesting why is there a difference between running the agent locally in container vs in k8s deployment then

Christopher Boyd · Answer

They would be using different infrastructure for execution locally it s using the process docker I d have to check I guess more specifically what logs do you want to come from your agent log

Christopher Boyd · Answer

The agent log is the traffic back and forth between Orion cloud and the executing infrastructure

Kirill Popov · Answer

< Christopher Boyd> the execution of the flow is always on KubernetesJob infra block in all 3 cases only agent execution environment changes

Christopher Boyd · Answer

So you want the flow logs

Kirill Popov · Answer

log wise i m only looking for status changes of flows amp tasks similar to the first quote streamed back to the agent at least now

Kirill Popov · Answer

amp i m not using the task flow loggers now but am getting the logs when running the agent locally or in local container

Christopher Boyd · Answer

Christopher Boyd · Answer

Are you executing locally or configured to use the cloud

Kirill Popov · Answer

< Christopher Boyd> i am using Prefect Cloud for job metadata orchestration kubernetes in digitalocean cloud for flow execution amp now want to use the same kubernetes for hosting the agent

Christopher Boyd · Answer

The info that you re missing from the pods flows are being emitted from the flows tasks themselves ```Task run My Example Task c06c9343 0 Finished in state Completed 19 41 52 443 | INFO | Flow run expert lyrebird Finished in state Completed ```

Christopher Boyd · Answer

This should be able to help but in short you need to include logging contexts into your flows

Christopher Boyd · Answer

Kirill Popov · Answer

thanks < Christopher Boyd> still confused as to why is there a difference where do I run the agent will try to see if adding explicit logging changes anything

Christopher Boyd · Answer

< Kirill Popov> when I add the logging to my flow this is what I get out of the actual pod job ```21 20 27 569 | INFO | Flow run gifted copperhead Created task run log platform info afea9710 0 for task log platform info 21 20 27 570 | INFO | Flow run gifted copperhead Executing log platform info afea9710 0 immediately 21 20 27 711 | INFO | Task run log platform info afea9710 0 Host s network name = gifted copperheadx62rw vz42r 21 20 27 712 | INFO | Task run log platform info afea9710 0 Python version = 3 9 15 21 20 27 713 | INFO | Task run log platform info afea9710 0 Platform information instance type = Linux 5 4 0 1085 azure x86 64 with glibc2 31 21 20 27 713 | INFO | Task run log platform info afea9710 0 OS Arch = linux x86 64 21 20 27 713 | INFO | Task run log platform info afea9710 0 Prefect Version = 2 6 3 21 20 27 714 | INFO | Task run log platform info afea9710 0 Prefect API Version = 0 8 2 21 20 27 780 | INFO | Task run log platform info afea9710 0 Finished in state Completed 21 20 27 900 | INFO | Flow run gifted copperhead Finished in state Completed All states completed ```

Christopher Boyd · Answer

This is what I get from the agent ```21 20 20 638 | INFO | prefect agent Completed submission of flow run b1988454 b944 4be4 ad4a affa22ea1dcf 21 20 20 745 | INFO | prefect infrastructure kubernetes job Job gifted copperheadx62rw Pod has status Pending 21 20 21 612 | INFO | prefect infrastructure kubernetes job Job gifted copperheadx62rw Pod has status Running ```