https://prefect.io logo
Join Slack
Powered by
Hi! I have a flow that executes a `RunNamespacedJo...
# prefect-community
a
Hi! I have a flow that executes a 
RunNamespacedJob
task. Because of a known bug the way I'm obtaining the logs are via 
ReadNamespacedPodLogs
. All that working well. Recently I changed the pod spec to run the application as an 
initContainer
then some docker image to capture results as the actual 
container
. So I have an image that generates content and logs in the 
ìnitContainer
and a 
container
that process the produced content (upload to gcs basically). Problem: the logs from the initContainer are not being captured by prefect, only the logs from the container. Any recommendation? 🙏
✅ 1
the flow, as you can see it's waiting for the pod to finish then get the logs from it, not reading the logs while running
this might be the reason, inside the python kubernetes client lib
Copy code
:param str container: The container for which to stream logs. Defaults to only container if there is one container in the pod.
I think at the end is going to be easier to have 2 different pods using a shared disk
a
This is interesting, I've never seen this pattern before to use init containers to run an actual application, I thought they are meant more for setup scripts before some other app can start. Is the only reason to use an init container here because init containers run until completion regardless of failure? I wonder whether this is something you can instead tackle using prefect triggers/state handlers, or with https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/?
👍 1
given that the logs from the actual container are captured, I wonder whether you may solve the problem by turning the app running in init container to a normal container and identify the problem behind why someone decided to run it in init container in the first place and solve it differently
a
I decided that :D Idea is we have dynamic flows that triggers random docker images. All I have is a docker image (and some k8s settings). That docker image is 100% independent of prefect, but still I want to run it within prefect. And here is the magic. I want to have a way to produce artifacts (ex: input payload to a http request, response, CSV, etc) then store those on GCS. Something I can do is say to the image "put your artifacts in this folder" but that's it there is no way to connect with prefect artifact controls.
Why initcontainer? Idea is to sync reading files from app execution, is not really a hard requirement, I found it good because of the "will run until completion before next step", so next step can read the files from a volume and upload whatever is there to GCS automatically
There could be other approaches tho
a
ahh I see, definitely a hard problem orchestrating processes that are completely independent and don't speak the same language but need to work in coordination and exchange data I asked our Kubernetes expert if he has some ideas, will let you know if he recommends sth
a
awesome, thank you!
🙌 1
a
I got a response and generally speaking we assume that jobs that we’re orchestrating only have one container, and it seems like a valid feature request to capture logs from all containers. One of the challenges is that each of the containers has its own log output and we’d need to aggregate them all, and some of them can run in parallel (e.g. multiple init containers or multiple containers). Due to the focus on Prefect 2.0, we don't have resources to make changes to this 1.0 task, but if you would want to contribute yourself or write custom logic for this in your flow, you could potentially add a flag to 
ReadNamespacedPodLogs
to specify container names and for the future, I can imagine integration like this in prefect-kubernetes Collection in 2.0
a
I'll give it a try 👍
🙌 1
a
nice work!
a
this is working amazingly well, thanks for your push! 👌
🙌 1
a
Awesome work getting this done success kid
3 Views
Open in Slack
PreviousNext
Powered by