Hi everyone is there an easy way to get the Docker container

Question

Hi everyone is there an easy way to get the Docker container id for a flow run We are currently get it by checking the prefect logs for the flow run but I m wondering if there s a better way We re using the Docker agent and we are on prefect 0 15 13

Anna Geller · Answer

why do you need a container ID

Anna Geller · Answer

not sure if we track it I honestly don t know your best bet would be to investigate Docker agent source code sorry I m not more helpful here

Lee Mendelowitz · Answer

We re having issues with prefect logging not capturing everything `prefect get logs id lt id gt ` doesn t return all of the output So we set up the Docker daemon to log to AWS CloudWatch Logs but those logs we must retrieve by container id

Lee Mendelowitz · Answer

The AWS CloudWatch Logs has logging output for flow runs that isn t returned by `prefect get logs` It s mysterious because we are using the prefect loggers this is output from a ShellTask with `stream output = True`

Anna Geller · Answer

why don t you leverage Prefect backend for logs the UI should display all the logs could you explain a bit more why did you decide to retrieve logs via CLI command

Lee Mendelowitz · Answer

So I could view or work with the entire log file easily `prefect get logs id lt id gt | less` or ``prefect get logs id lt id gt | grep` is easier than clicking thru the UI which for me is only displaying a small portion of the log file It could be a largish log file 1 5 MB

Anna Geller · Answer

it s definitely your choice how you decide to do it if the UI doesn t show all logs that you need this seems to be a different problem though

Anna Geller · Answer

what is your end goal is it only about inspecting logs yourself from CLI UI or do you try to retrieve those for some different purpose

Lee Mendelowitz · Answer

It s about inspecting logs myself the flow run failed with an error and I d like to take a closer look at the logging

Anna Geller · Answer

gotcha I ve never used that feature myself but if this doesn t show all logs you need I believe this is the actual problem here rather than logging the container ID

Anna Geller · Answer

your flow run logs should be collected on the backend and whether you retrieve it from UI or CLI it should give you the same logs

Lee Mendelowitz · Answer

yeah I agree that s the real problem here For reference retrieving the logs from AWS CloudWatch logs for this flow runs returns a log file of about 8 MB but `prefect get logs` returns a log of just 412kb Maybe `prefect get logs` has some limit of how many log statements it will return or something like that

Anna Geller · Answer

Checking after holidays LMK if this is still an open issue looks like the issue you had so far was that some logs did not get properly stored in the backend if this is still an ongoing issue you would need to cross check how you add logs in your flow how you configured the logger itself e g log level and ideally share some small example flow that I could reproduce that shows the issue