@Marvin how do I process/save logs that are being printed from my container, with my current solution they are being printed but not captures in the eavesdropper, is there another way I can access them? @task(name="Parse Jora Jobs", retries=0, log_prints=True) def run_job_parser(): logger = get_run_logger() with LogEavesdropper("my_logger") as eavesdropper: logging.getLogger("my_logger").info("Hello, world!") logging.getLogger("my_logger.child_module").info("Another one!") docker_container_block = DockerContainer.load("mediariot-docker-parser") container_result = docker_container_block.run() logging.getLogger("my_logger.child_module").info("Another another one!") print(eavesdropper.text())

To capture and process logs from a Docker container in Prefect, you can use the

get_docker_container_logs

method. This method retrieves the logs from the container, similar to the

docker logs

command. Below is how you might adapt your code to use this: 1. Modify Your Task to Retrieve Container Logs:

python
from prefect import task, flow, get_run_logger
from prefect_docker import DockerContainer
from prefect_docker.containers import get_docker_container_logs

@task(name="Parse Jora Jobs", retries=0, log_prints=True)
def run_job_parser():
    logger = get_run_logger()
    logging.getLogger("my_logger").info("Hello, world!")
    logging.getLogger("my_logger.child_module").info("Another one!")

    # Load and run the Docker container
    docker_container_block = DockerContainer.load("mediariot-docker-parser")
    container_result = docker_container_block.run()

    # Get logs from the running container
    container_id = container_result.id
    logs = get_docker_container_logs(container_id)

    logging.getLogger("my_logger.child_module").info("Container logs:\n" + logs)
    logging.getLogger("my_logger.child_module").info("Another another one!")

    # Print or process logs
    print(logs)

# Example flow
@flow
def example_flow():
    run_job_parser()

if __name__ == "__main__":
    example_flow()

In this updated code: 1. The

get_docker_container_logs

method is used to retrieve logs directly from the container. 2. These logs can be processed or printed as needed within your Prefect task. For additional details, you can refer to the Prefect Docker integration documentation: https://docs.prefect.io/latest/integrations/prefect-docker/containers/ For more tips on handling logging in Prefect, you can check out the Prefect logging guide: https://docs.prefect.io/latest/guides/logs/ And here are some relevant GitHub issues for more context: - Containers are kept open when they crash: https://github.com/PrefectHQ/prefect/issues/7560 - Logs-related improvements: https://github.com/PrefectHQ/prefect/issues/8557 Yes, I know. It never ends.

@Marvin yes but then I have to use the prefect_docker pip package, which is not available inside workers. I guess I can install it but it feels like a workaround where there should be a better/different solution

It looks like there isn't a ready-made solution in the Prefect documentation to avoid using the

prefect_docker

package for logging from a Docker container. However, you could use Python's

docker

library directly to get the logs from your Docker container. Here's a suggestion on how you can do it: 1. Install the Docker SDK for Python:

sh
    pip install docker

2. Modify Your Code to Retrieve Docker Logs using the Docker SDK:

python
from prefect import task, flow, get_run_logger
import docker
import logging

@task(name="Parse Jora Jobs", retries=0, log_prints=True)
def run_job_parser():
    logger = get_run_logger()
    logging.getLogger("my_logger").info("Hello, world!")
    logging.getLogger("my_logger.child_module").info("Another one!")

    # Initialize a Docker client
    client = docker.from_env()

    # Run your Docker container
    container = client.containers.run("your_image_name", "command", detach=True)

    # Wait for the container to finish its task
    container.wait()

    # Retrieve and log the container logs
    logs = container.logs().decode("utf-8")
    logging.getLogger("my_logger.child_module").info("Container logs:\n" + logs)
    logging.getLogger("my_logger.child_module").info("Another another one!")

    # Print or process logs
    print(logs)

# Example flow
@flow
def example_flow():
    run_job_parser()

if __name__ == "__main__":
    example_flow()

### Additional Resources: - Prefect logging guide: https://docs.prefect.io/latest/guides/logs/ This way, you avoid needing the

prefect_docker

package and can directly work with the Docker SDK to capture logs from your containers. Yes, I know, it's still more to do. No rest for the androids.