https://prefect.io logo
Title
k

Kirill Popov

10/28/2022, 8:22 PM
hey folks, looking for some help here with setting up prefect 2 on K8S When running an agent in kubernetes, the agent does not receive logs from workers... among other things, this means I can not see if flow run succeeded/failed from agent logs Here is my setup: • all flows are stored in S3 • all flows run on the same infra block which is a KubernetesJob - new pod is spun up for each flow • using Prefect Cloud to orchestrate Everything was great when i was running prefect agent locally -- agent produces logs (output in CLI) similar to:
19:41:45.583 | INFO    | prefect.agent - Submitting flow run 'd03564b9-ad4e-4310-8662-184616d6406f'
19:41:47.416 | INFO    | prefect.agent - Completed submission of flow run 'd03564b9-ad4e-4310-8662-184616d6406f'
19:41:48.471 | INFO    | prefect.infrastructure.kubernetes-job - Job 'expert-lyrebird-xjq42': Pod has status 'Pending'.
19:41:49.155 | INFO    | prefect.infrastructure.kubernetes-job - Job 'expert-lyrebird-xjq42': Pod has status 'Running'.
19:41:52.098 | INFO    | Flow run 'expert-lyrebird' - Created task run 'My Example Task-c06c9343-0' for task 'My Example Task'
19:41:52.098 | INFO    | Flow run 'expert-lyrebird' - Executing 'My Example Task-c06c9343-0' immediately...
19:41:52.345 | INFO    | Task run 'My Example Task-c06c9343-0' - Finished in state Completed()
19:41:52.443 | INFO    | Flow run 'expert-lyrebird' - Finished in state Completed()
When running the agent locally but in docker container, the logs look similar. Now I want the agent to run in a kubernetes deployment to automatically manage its uptime. Only 1 replica of agent pod is used. However I only receive part of the log in the agent cli output -- it looks like no info is coming back from the worker pods back to the agent pod...
19:59:58.687 | INFO    | prefect.agent - Submitting flow run 'a510b13d-b0b5-4067-8610-518e6a2b45a8'
19:59:59.268 | INFO    | prefect.agent - Completed submission of flow run 'a510b13d-b0b5-4067-8610-518e6a2b45a8'
19:59:59.305 | INFO    | prefect.infrastructure.kubernetes-job - Job 'secret-iguana-zlm4f': Pod has status 'Pending'.
20:00:01.349 | INFO    | prefect.infrastructure.kubernetes-job - Job 'secret-iguana-zlm4f': Pod has status 'Running'.
What is going on ? how can I make the agent running on K8S produce a complete log ? let me know if you have any ideas -- I am fairly new to k8s so might be something basic, but looks unexpected to me
👀 1
⁉️ 1
c

Christopher Boyd

10/28/2022, 8:32 PM
I think if you want logs from your jobs, you need to add logging contexts to your flow
This is expected behavior
You need a task logger / flow logger in the flow to emit logs from the jobs /pods
🧠 1
k

Kirill Popov

10/28/2022, 8:33 PM
@Christopher Boyd ok interesting…. why is there a difference between running the agent locally/in container vs in k8s deployment then?
c

Christopher Boyd

10/28/2022, 8:35 PM
They would be using different infrastructure for execution, locally it’s using the process, docker I’d have to check - I guess more specifically what logs do you want to come from your agent log
The agent log is the traffic back and forth between Orion/ cloud , and the executing infrastructure
k

Kirill Popov

10/28/2022, 8:37 PM
@Christopher Boyd the execution of the flow is always on KubernetesJob infra block in all 3 cases, only agent execution environment changes
c

Christopher Boyd

10/28/2022, 8:37 PM
So you want the flow logs ?
k

Kirill Popov

10/28/2022, 8:37 PM
log-wise i’m only looking for status changes of flows & tasks similar to the first quote streamed back to the agent (at least now)
& i’m not using the task/flow loggers now but am getting the logs when running the agent locally or in local container
c

Christopher Boyd

10/28/2022, 8:39 PM
Are you executing locally or configured to use the cloud
k

Kirill Popov

10/28/2022, 8:41 PM
@Christopher Boyd i am using Prefect Cloud for job metadata/orchestration, kubernetes in digitalocean cloud for flow execution & now want to use the same kubernetes for hosting the agent
c

Christopher Boyd

10/28/2022, 8:46 PM
The info that you’re missing from the pods / flows, are being emitted from the flows / tasks themselves:
Task run 'My Example Task-c06c9343-0' - Finished in state Completed()
19:41:52.443 | INFO    | Flow run 'expert-lyrebird' - Finished in state Completed()
This should be able to help, but in short, you need to include logging contexts into your flows
k

Kirill Popov

10/28/2022, 8:55 PM
thanks @Christopher Boyd still confused as to why is there a difference where do I run the agent, will try to see if adding explicit logging changes anything
c

Christopher Boyd

10/28/2022, 9:21 PM
@Kirill Popov - when I add the logging to my flow, this is what I get out of the actual pod / job:
21:20:27.569 | INFO    | Flow run 'gifted-copperhead' - Created task run 'log_platform_info-afea9710-0' for task 'log_platform_info'
21:20:27.570 | INFO    | Flow run 'gifted-copperhead' - Executing 'log_platform_info-afea9710-0' immediately...
21:20:27.711 | INFO    | Task run 'log_platform_info-afea9710-0' - Host's network name = gifted-copperheadx62rw-vz42r
21:20:27.712 | INFO    | Task run 'log_platform_info-afea9710-0' - Python version = 3.9.15
21:20:27.713 | INFO    | Task run 'log_platform_info-afea9710-0' - Platform information (instance type) = Linux-5.4.0-1085-azure-x86_64-with-glibc2.31 
21:20:27.713 | INFO    | Task run 'log_platform_info-afea9710-0' - OS/Arch = linux/x86_64
21:20:27.713 | INFO    | Task run 'log_platform_info-afea9710-0' - Prefect Version = 2.6.3 🚀
21:20:27.714 | INFO    | Task run 'log_platform_info-afea9710-0' - Prefect API Version = 0.8.2
21:20:27.780 | INFO    | Task run 'log_platform_info-afea9710-0' - Finished in state Completed()
21:20:27.900 | INFO    | Flow run 'gifted-copperhead' - Finished in state Completed('All states completed.')
This is what I get from the agent:
21:20:20.638 | INFO    | prefect.agent - Completed submission of flow run 'b1988454-b944-4be4-ad4a-affa22ea1dcf'
21:20:20.745 | INFO    | prefect.infrastructure.kubernetes-job - Job 'gifted-copperheadx62rw': Pod has status 'Pending'.
21:20:21.612 | INFO    | prefect.infrastructure.kubernetes-job - Job 'gifted-copperheadx62rw': Pod has status 'Running'.