Prefect 1.0 - Is there a recommended way to reroute/duplicate the pip logs for

EXTRA_PIP_PACKAGES

to Cloud UI? When using Kubernetes agent, those logs only show up in job pods logs and are not available in Cloud UI.

Hey Shaoyi!! I would recommend looking at this documentation about adding external loggers in 1.0: https://docs-v1.prefect.io/core/concepts/logging.html#extra-loggers You should be able to add loggers from external sources and those should appear in the Cloud UI

Hey Rob! I understand the external logger configuration, but I think the

pip install

happens outside of the python flow and executed as a shell command. https://github.com/PrefectHQ/prefect/blob/main/scripts/entrypoint.sh#L10 That means those logs go to stdout directly, right? How would I reroute the pip install logs using the external logger configuration?

Why is this important for you to log that information? Is this to print some version or check whether a package has been installed? You can log the output of pip show or log package version to satisfy the same objective

When using EXTRA_PIP_PACKAGES with Kuberentes agent, if the pip install fails, there’s no way for users to see the actual failure from Cloud UI. Only people with access to the Kubernetes cluster can view the job pod logs.

In that case I would recommend tackling the root cause of the problem which is packaging -- the EXTRA_PIP_PACKAGES is meant only for getting started, I think for production it would be better to build your own Docker image and reference that instead of installing packages at runtime

As a software engineer, I hate EXTRA_PIP_PACKAGES and I never use it, even for getting started. However, data analysts like that and some of them have no idea of Docker and Kuberentes.

Exactly!