https://prefect.io logo
Title
m

Mukamisha jocelyne

03/12/2022, 11:22 AM
Hi! I am trying to create a directory within a container, something simple that looks like the code snippet below (in the thread). The task is running successfully but failing to create the data directory. I was thinking that it's supposed to create the directory without any issue, any suggestions to what I am missing please? Thank you
k

Kevin Kho

03/12/2022, 4:11 PM
Hi @Mukamisha jocelyne, how are you checking if the folder is created? When you get the chance, could you move the code snippet into this thread to keep the main channel cleaner?
m

Mukamisha jocelyne

03/12/2022, 5:27 PM
FLOW_NAME = "docker_script"
docker_storage = Docker(
    image_name="test",
    image_tag="latest",
    stored_as_script=True,
    path=f"/app/test3.py",
)


@task(log_stdout=True)
def create_directory():
    text = f"..."
    ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
    local_path = os.path.join(ROOT_DIR, "data")

    Path(local_path).mkdir(parents=True, exist_ok=True)
    return text

with Flow(
    FLOW_NAME,
    storage=docker_storage,
    run_config=DockerRun(
        image="test:latest",
        labels=["docker"],
    ),
) as flow:
    create_dir = create_directory()

if __name__ == "__main__":
    docker_storage.add_flow(flow)
    flow.register(project_name="test_project", build=False)
Sorry about that, I am checking it inside the container created. When running the script without using prefect flow and tasks , I can see the folder being created inside the container but once I use this approach above it doesn't work.
k

Kevin Kho

03/12/2022, 7:09 PM
No worries I think it would work. I’m more of asking if you are checking using
docker exec
rather than Docker run?
m

Mukamisha jocelyne

03/12/2022, 7:40 PM
oh okay, I checked using both
k

Kevin Kho

03/12/2022, 7:48 PM
Docker exec on the new container doesn’t show the new directory? You can this with a Docker agent?
m

Mukamisha jocelyne

03/12/2022, 7:58 PM
No the docker exec doesn't show it
I didn't get the 2nd qsn you asked
k

Kevin Kho

03/12/2022, 7:59 PM
Sorry typo. You ran* with a Docker agent?
I’ll test this quickly now
m

Mukamisha jocelyne

03/12/2022, 8:00 PM
Oh yes I did run with the docker agent
k

Kevin Kho

03/12/2022, 8:14 PM
A bit confused. Isn’t the container cleaned up after Flow execution? How do you exec into it?
m

Mukamisha jocelyne

03/12/2022, 8:25 PM
Is the container supposed to clean up? I'm not sure. I opened the terminal on the side and exec into the container produced by the image
k

Kevin Kho

03/12/2022, 8:31 PM
I think so. What Prefect version are you on?
I am also not sure storing stuff in the Docker container is common practice. We should add a volume maybe and persist data that way. Could you tell me more about the use case here?
m

Mukamisha jocelyne

03/12/2022, 9:16 PM
oh okay, if the container cleans up then that would make sense, i'm using prefect:0.15.12. I was trying create a directory to store some models artifacts pulled down from s3 for generating encodings of images after producing those encodings, I would then store the results on s3.
Thank you, I think I will use volumes for now
k

Kevin Kho

03/12/2022, 11:34 PM
I think you can just use s3 directly in one flow instead of using a volume then?