https://prefect.io logo
Title
s

Shaoyi Zhang

08/30/2022, 5:51 PM
Prefect 1.0 - Is there a recommended way to reroute/duplicate the pip logs for
EXTRA_PIP_PACKAGES
to Cloud UI? When using Kubernetes agent, those logs only show up in job pods logs and are not available in Cloud UI.
r

Rob Freedy

08/30/2022, 7:17 PM
Hey Shaoyi!! I would recommend looking at this documentation about adding external loggers in 1.0: https://docs-v1.prefect.io/core/concepts/logging.html#extra-loggers You should be able to add loggers from external sources and those should appear in the Cloud UI
s

Shaoyi Zhang

08/30/2022, 7:25 PM
Hey Rob! I understand the external logger configuration, but I think the
pip install
happens outside of the python flow and executed as a shell command. https://github.com/PrefectHQ/prefect/blob/main/scripts/entrypoint.sh#L10 That means those logs go to stdout directly, right? How would I reroute the pip install logs using the external logger configuration?
a

Anna Geller

08/31/2022, 8:14 PM
Why is this important for you to log that information? Is this to print some version or check whether a package has been installed? You can log the output of pip show or log package version to satisfy the same objective
You could open an issue but given it's 1.0 I would not expect anyone working on this. We could check how we can log that information better in 2.0 though
s

Shaoyi Zhang

08/31/2022, 8:19 PM
When using EXTRA_PIP_PACKAGES with Kuberentes agent, if the pip install fails, there’s no way for users to see the actual failure from Cloud UI. Only people with access to the Kubernetes cluster can view the job pod logs.
a

Anna Geller

08/31/2022, 9:32 PM
In that case I would recommend tackling the root cause of the problem which is packaging -- the EXTRA_PIP_PACKAGES is meant only for getting started, I think for production it would be better to build your own Docker image and reference that instead of installing packages at runtime
s

Shaoyi Zhang

08/31/2022, 9:36 PM
As a software engineer, I hate EXTRA_PIP_PACKAGES and I never use it, even for getting started. However, data analysts like that and some of them have no idea of Docker and Kuberentes.
Thank you for your suggestions though. I think the walkaround will be internal tooling. For example, we can process the flow code in CI/CD and explicitly install packages from within the flow instead of relying on the entrypoint.sh from Prefect official image.
a

Anna Geller

08/31/2022, 10:50 PM
Exactly!