https://prefect.io logo
Title
s

scott

02/06/2023, 2:54 PM
prefect agents running on our VM kick off flows, all of which have tasks that execute shell commands that spin up docker containers. We have an issue with two flows that use parallelization inside the container they’re running in using the R programming language. These two flows run just fine if we run the docker containers outside of a prefect context - but when these flows are triggered by a prefect agent the flows start but then hang after a while (e.g., 30 min to 1 hr) with no logs returned, no errors, just hanging. It seems a little surprising if this was due to the prefect python client, but that seems to be a likely source of the problem. Any ideas? Anyone else run into this?
m

Mason Menges

02/06/2023, 5:41 PM
Hey Scott do you see any errors on the prefect agent in this case? if you haven't already setting the logging level to Debug on the agent may also help narrow down where it's getting stuck, if you're not seeing any logs I'd wager a guess the container isn't even starting so the agent logs would likely be more informative in this case as to what might be occurring
s

scott

02/06/2023, 5:48 PM
Thanks, I don’t know about prefect agent errors. But good idea about setting logging level to Debug. Can setting logging level to Debug be done with the command line interface?
Doesn’t look like it from
prefect agent start --help
c

Christopher Boyd

02/07/2023, 1:50 PM
Prefect config set PREFECT_LOGGING_LEVEL=DEBUG
s

scott

02/07/2023, 2:41 PM
Thanks @Christopher Boyd - What if I’m running agents with
supervisor
? Still use
prefect config set
approach, or set the env var with supervsior?
c

Christopher Boyd

02/07/2023, 2:54 PM
It should pick up environment variables so that way works as well
s

scott

02/07/2023, 4:38 PM
ok thanks