@Marvin is it possible to return any data directly from a docker container that is run through DockerContainer.run()

You can indeed retrieve data from a Docker container when using the

DockerContainer

infrastructure in Prefect. Here’s a summary of options you have: 1. Logging: Prefect provides the

get_docker_container_logs

method to fetch logs from the container, similar to the

docker logs

command. - Source: Prefect Docker Containers Integration Docs 2. Environment Variables: You can pass environment variables into your Docker container which can then be used by your application inside the container. - Source: Prefect Docker Guide 3. Direct Interaction via Prefect Tasks: Use Prefect tasks to capture and return the output directly from the Docker container.

python
   from prefect import flow, task
   from prefect_docker import DockerContainer
   
   @task
   def run_docker_container():
       container = DockerContainer(
           image="prefecthq/prefect",
           command="echo 'hello world!'"
       )
       logs = container.get_docker_container_logs()
       return logs
   
   @flow
   def docker_flow():
       result = run_docker_container()
       print(result)

   docker_flow()

This example creates and runs a Docker container, retrieves its logs, and returns them. 4. Bug Reports and Issues: There are ongoing discussions and examples in the Prefect GitHub repository which may provide context or additional options. - GitHub Issue Example: Using

RemoteFileSystem

storage with

docker-container

infrastructure. - Source: Prefect GitHub Issue 9159 Using one or a combination of the above methods, you can effectively capture return data from a Docker container run via the

DockerContainer

infrastructure in Prefect. Feel free to explore the provided links for more detailed documentation and discussion threads.

the way I run it there is no "get_docker_container_logs": docker_container_block = DockerContainer.load("mediariot-docker-parser") result = docker_container_block.run() print(f"result = {result}") logs_result = docker_container_block.get_docker_container_logs()

File "/mnt/nas/shared/projects/jobify/workflows/job_etl.py", line 15, in run_job_parser                                                                                                                 
    logs_result = docker_container_block.get_docker_container_logs()                                                                                                                                      
AttributeError: 'DockerContainer' object has no attribute 'get_docker_container_logs'

It appears that the

get_docker_container_logs

method might not be available in your current implementation of the

DockerContainer

block. This discrepancy might be due to the version of Prefect or differing configurations. Here is what you can try next: 1. Check Prefect Version: Ensure you are using the latest version of Prefect where such methods might be available.

bash
   pip install --upgrade prefect

2. Use Prefect’s Task Interface: If

get_docker_container_logs

is not present, consider using Prefect’s Docker Container interface capabilities directly in a robust way.

python
   from prefect import task, flow
   from prefect.infrastructure.docker import DockerContainer

   @task
   def run_container_and_get_logs():
       docker_container_block = DockerContainer.load("mediariot-docker-parser")
       result = docker_container_block.run()
       logs = result.get("logs")  # Example of fetching logs after running the container
       return logs

   @flow
   def docker_flow():
       logs = run_container_and_get_logs()
       print(f"Container Logs: {logs}")

   docker_flow()

3. Update Docker Container Block Implementation: If the method indeed should exist, it might be a matter of updating the Prefect library or extending the Docker block to include such a method.

python
   from prefect.infrastructure.docker import DockerContainer as BaseDockerContainer

   class DockerContainerWithLogs(BaseDockerContainer):
       async def get_docker_container_logs(self, container_id: str):
           # Custom implementation or using existing Docker SDK
           import docker
           client = docker.from_env()
           container = client.containers.get(container_id)
           logs = container.logs()
           return logs

   docker_container_block = DockerContainerWithLogs.load("mediariot-docker-parser")
   result = docker_container_block.run()
   logs_result = docker_container_block.get_docker_container_logs(result['container_id']) 
   print(f"logs_result: {logs_result}")

For further details, see the Prefect Docker Containers Integration documentation: * https://docs.prefect.io/latest/integrations/prefect-docker/containers/ If none of these solutions work, feel free to raise an issue on the Prefect GitHub repository with details about the error: * Prefect GitHub Issues