Tim-Oliver11/14/2022, 1:57 PM
Tim Galvin11/14/2022, 2:00 PM
and I am seeing logs produced in my tasks (provided a
is used to get an appropriate logger) in the prefect UI and flow run without any issues
Tim-Oliver11/14/2022, 2:12 PM
as executor and the logs are not getting back from the workers.
(on the local machine) works.
Tim Galvin11/14/2022, 3:01 PM
where a singularity command was issued. Said command would run an MPI application within the container, and an appropriate
would get the command to run across many compute nodes. This command produced a large set of outputs on
that I would capture through the normal
result object, and then manually
them. I found that these logs were not captured often, but everything ran successfully. Eventually I converged towards having a
type command just after my my
command. When I did that my missing logs were always captured appropriately. My hunch is a simple one - the process managing the main orion server does not persist long enough to retrieve all outputs. This subprocess sleep command should block long enough for this exchange to carry out. At the time I worked through this I was not managing my own orion prefect server - I was relying on the main process to fire one up for the lifetime of the flow.
Tim-Oliver11/14/2022, 4:19 PM
. I tested it without
and then the logs get properly through as you said.