Alvaro Durán Tovar

    Alvaro Durán Tovar

    2 months ago
    Hi! I have a flow that executes a
    RunNamespacedJob
    task. Because of a known bug the way I'm obtaining the logs are via
    ReadNamespacedPodLogs
    . All that working well. Recently I changed the pod spec to run the application as an
    initContainer
    then some docker image to capture results as the actual
    container
    . So I have an image that generates content and logs in the
    ìnitContainer
    and a
    container
    that process the produced content (upload to gcs basically). Problem: the logs from the initContainer are not being captured by prefect, only the logs from the container. Any recommendation? 🙏
    the flow, as you can see it's waiting for the pod to finish then get the logs from it, not reading the logs while running
    this might be the reason, inside the python kubernetes client lib
    :param str container: The container for which to stream logs. Defaults to only container if there is one container in the pod.
    I think at the end is going to be easier to have 2 different pods using a shared disk
    Anna Geller

    Anna Geller

    2 months ago
    This is interesting, I've never seen this pattern before to use init containers to run an actual application, I thought they are meant more for setup scripts before some other app can start. Is the only reason to use an init container here because init containers run until completion regardless of failure? I wonder whether this is something you can instead tackle using prefect triggers/state handlers, or with https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/?
    given that the logs from the actual container are captured, I wonder whether you may solve the problem by turning the app running in init container to a normal container and identify the problem behind why someone decided to run it in init container in the first place and solve it differently
    Alvaro Durán Tovar

    Alvaro Durán Tovar

    2 months ago
    I decided that 😄 Idea is we have dynamic flows that triggers random docker images. All I have is a docker image (and some k8s settings). That docker image is 100% independent of prefect, but still I want to run it within prefect. And here is the magic. I want to have a way to produce artifacts (ex: input payload to a http request, response, CSV, etc) then store those on GCS. Something I can do is say to the image "put your artifacts in this folder" but that's it there is no way to connect with prefect artifact controls.
    Why initcontainer? Idea is to sync reading files from app execution, is not really a hard requirement, I found it good because of the "will run until completion before next step", so next step can read the files from a volume and upload whatever is there to GCS automatically
    There could be other approaches tho
    Anna Geller

    Anna Geller

    2 months ago
    ahh I see, definitely a hard problem orchestrating processes that are completely independent and don't speak the same language but need to work in coordination and exchange data I asked our Kubernetes expert if he has some ideas, will let you know if he recommends sth
    Alvaro Durán Tovar

    Alvaro Durán Tovar

    2 months ago
    awesome, thank you!
    Anna Geller

    Anna Geller

    2 months ago
    I got a response and generally speaking we assume that jobs that we’re orchestrating only have one container, and it seems like a valid feature request to capture logs from all containers. One of the challenges is that each of the containers has its own log output and we’d need to aggregate them all, and some of them can run in parallel (e.g. multiple init containers or multiple containers). Due to the focus on Prefect 2.0, we don't have resources to make changes to this 1.0 task, but if you would want to contribute yourself or write custom logic for this in your flow, you could potentially add a flag to
    ReadNamespacedPodLogs
    to specify container names and for the future, I can imagine integration like this in prefect-kubernetes Collection in 2.0
    Alvaro Durán Tovar

    Alvaro Durán Tovar

    2 months ago
    I'll give it a try 👍
    Anna Geller

    Anna Geller

    2 months ago
    nice work!
    Alvaro Durán Tovar

    Alvaro Durán Tovar

    2 months ago
    this is working amazingly well, thanks for your push! 👌
    Anna Geller

    Anna Geller

    2 months ago
    Awesome work getting this done 😒uccess-kid: