https://prefect.io logo
Title
s

scott

02/10/2023, 5:14 PM
All of our Prefect tasks run shell commands using
prefect-shell
(we’re currently moving from using
shell_run_command
to
ShellOperation
). These shell commands all spin up docker containers. Sometimes a container will hang without any logs for up to 24 hrs. The only thing we’ve found to work is “tickling” the process connected to the container (we’ve done e..g,
tail /proc/<pid>/fd/2
), and then we get logs again. We haven’t seen this problem when we start docker containers outside of a prefect context, so we’re thinking it’s something to do with prefect. it is possible this will go away as we transition to
ShellOperation
, but perhaps not. Curious if anyone has seen this behavior too? (related thread https://prefect-community.slack.com/archives/C048ZHT5U3U/p1675282219967509 )