Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hello all, I had a question regarding logging: I have a flow that largely looks like the one outlined here: <https://docs.prefect.io/core/examples/functional_docker.html> that we run through the Server UI. I have noticed that the logs from the container only show up if there is a problem with the code and I would like to know if there is a way in the GetContainerLogs  call that we can collect the logs from the python standard logging library and have them printed in the UI for ease of use. Currently info, debug, and exceptions get logged to disk and I would like to have both; logged to disk and shown in the UI

Hi <@U01HPF1476W>!

Are you using the `GetContainerLogs` Task from the Task Library?

Hi <@UKVFX6N3B>, yes I am. My flow is currently almost identical to what is shown in the functional docker example.

Under the hook I believe that we’re using this library: <https://docker-py.readthedocs.io/en/stable/containers.html>

You may be able to pass a `log_level` kwarg to the Task definition and get all logs from the container

<@UKVFX6N3B> so I would need to configure the logging in the docker image to use Docker's logging, or am I able to still use python's logging module and return that info in the GetContainerLogs

Looks like it, though? <https://docs.docker.com/config/containers/logging/configure/>

This doc refers to the doc above <https://docker-py.readthedocs.io/en/stable/api.html#docker.types.LogConfig>

So it looks like this Task is working based on the docker logging config

Depending on what is happening inside your containers, could could also orchestrate that with Prefect and you would get access to the logging for free! If you’re running non-python processes, take a look at: <https://docs.prefect.io/api/latest/tasks/shell.html>

<@UKVFX6N3B> Hi Dylan, the docker container is python code that performs some ETL tasks and ideally I would have prefect orchestrate them all for the free logging; however I was unable to come up with a way of doing so that does not involve prefect re-building the image/increment flow version upon every minor change to the tasks as where it is being deployed to is very CPU limited. If you have insight on how I can have Prefect build the image 1 time unless there are major changes, i.e., to dependencies in the requirements.txt file, I would love to see an example so that I can easily have logs streamed to the Server UI