https://prefect.io logo
Join Slack
Powered by
Hi everyone - is there an easy way to get the Dock...
# prefect-community
l
Hi everyone - is there an easy way to get the Docker container id for a flow run? We are currently get it by checking the prefect logs for the flow run, but I’m wondering if there’s a better way. We’re using the Docker agent and we are on prefect 0.15.13.
a
why do you need a container ID?
not sure if we track it, I honestly don't know - your best bet would be to investigate Docker agent source code - sorry I'm not more helpful here
l
We’re having issues with prefect logging not capturing everything, 
prefect get logs --id <id>
doesn’t return all of the output. So we set up the Docker daemon to log to AWS CloudWatch Logs, but those logs we must retrieve by container id
The AWS CloudWatch Logs has logging output for flow runs that isn’t returned by 
prefect get logs
. It’s mysterious because we are using the prefect loggers…this is output from a ShellTask with 
stream_output = True
a
why don't you leverage Prefect backend for logs? the UI should display all the logs - could you explain a bit more why did you decide to retrieve logs via CLI command?
l
So I could view or work with the entire log file easily. 
prefect get logs --id <id> | less
or ``prefect get logs --id <id> | grep` is easier than clicking thru the UI, which for me is only displaying a small portion of the log file. It could be a largish log file, 1-5 MB
a
it's definitely your choice how you decide to do it - if the UI doesn't show all logs that you need, this seems to be a different problem though
what is your end goal? is it only about inspecting logs yourself from CLI/UI? or do you try to retrieve those for some different purpose?
l
It’s about inspecting logs myself - the flow run failed with an error and I’d like to take a closer look at the logging
a
gotcha - I've never used that feature myself but if this doesn't show all logs you need, I believe this is the actual problem here, rather than logging the container ID
your flow run logs should be collected on the backend and whether you retrieve it from UI or CLI, it should give you the same logs
l
yeah I agree that’s the real problem here. For reference, retrieving the logs from AWS CloudWatch logs for this flow runs returns a log file of about 8 MB, but 
prefect get logs
returns a log of just 412kb. Maybe 
prefect get logs
has some limit of how many log statements it will return or something like that
a
Checking after holidays: LMK if this is still an open issue looks like the issue you had so far was that some logs did not get properly stored in the backend - if this is still an ongoing issue, you would need to cross-check how you add logs in your flow, how you configured the logger itself (e.g. log level) and ideally share some small example flow that I could reproduce that shows the issue
6 Views
Open in Slack
PreviousNext
Powered by