This message was deleted Prefect Community #ask-marvin

Join Slack

This message was deleted.

# ask-marvin

Slackbot

07/02/2024, 11:28 AM

This message was deleted.

Marvin

07/02/2024, 11:28 AM

It looks like there's not much direct documentation or examples on specifically tailing all logs from a Docker container within a flow. However, here's a general approach you can use in Python to continuously fetch logs from a Docker container, which you can adapt for use in your Prefect flow: 1. Install Docker SDK if you haven't already:

Copy code

bash
pip install docker

2. Tail Logs in a Prefect Flow: Here's a simplified example:

Copy code

python
from prefect import flow, task
import docker

@task
def tail_logs(container_name: str):
    client = docker.from_env()
    container = client.containers.get(container_name)
    
    # This will continuously fetch logs
    for log in container.logs(stream=True):
        print(log.strip())

@flow
def docker_logs_flow(container_name: str):
    tail_logs(container_name)

if __name__ == "__main__":
    # Replace 'your_container_name' with the actual name of your Docker container
    docker_logs_flow("your_container_name")

This approach uses the Docker SDK for Python to stream logs continuously from the specified container. For more information, you can refer to the Docker SDK documentation here: https://docker-py.readthedocs.io/en/stable/ Remember, always consult the latest Prefect documentation and the Docker SDK guides to ensure compatibility and best practices. If you encounter issues with Prefect flows or logging, it’s also beneficial to check GitHub issues or the Prefect documentation for any updates or community solutions. Here’s the GitHub repository for Prefect: https://github.com/PrefectHQ/prefect

Marvin

07/02/2024, 11:38 AM

Your message was too long by 569 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

here if my flow:

import asyncio from prefect import flow, task, get_run_logger from prefect_docker.images import pull_docker_image from prefect_docker.containers import create_docker_container, start_docker_container, get_docker_container_logs from prefect_docker.credentials import DockerRegistryCredentials @task async def pull_image(credentials): logger = get_run_logger() # Pull the latest image from the private repository logger.info("Pulling the latest image from the private repository...") image = await pull_docker_image( repository="seedooinsights/build", tag="159d211", docker_registry_credentials=credentials ) logger.info(f"Pulled image: {image}") return image @task async def create_container(): logger = get_run_logger() # Create a container with the specified flags logger.info("Creating a container with the specified flags...") container = await create_docker_container( image="seedooinsights/build:159d211", command=["bash", "-c", "cd doloop/seedoo/indexing/preprocessing && bash indexer.sh /seedoodata/datasets/data/traffic_lights/data.pkl ./aaa traffic"], environment=[ "MKL_NUM_THREADS=8", "NUMBA_NUM_THREADS=8", "

Marvin

07/02/2024, 11:39 AM

Your message was too long by 477 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

here if my flow:

import asyncio from prefect import flow, task, get_run_logger from prefect_docker.images import pull_docker_image from prefect_docker.containers import create_docker_container, start_docker_container, get_docker_container_logs from prefect_docker.credentials import DockerRegistryCredentials @task async def pull_image(credentials): logger = get_run_logger() # Pull the latest image from the private repository logger.info("Pulling the latest image from the private repository...") image = await pull_docker_image( repository="seedooinsights/build", tag="159d211", docker_registry_credentials=credentials ) logger.info(f"Pulled image: {image}") return image @task async def create_container(): logger = get_run_logger() # Create a container with the specified flags logger.info("Creating a container with the specified flags...") container = await create_docker_container( image="seedooinsights/build:159d211", command=["bash", "-c", "cd doloop/seedoo/indexing/preprocessing && bash indexer.sh /seedoodata/datasets/data/traffic_lights/data.pkl ./aaa traffic"], volumes=[ "/seedoodata:/seedoodata", ], device_requests=[ { 'Driver': 'n

Open in Slack

Previous Next