https://prefect.io
Join Slack
Hi! I am trying to create a directory within a container, something simple that looks like the code ...
# ask-community
m
Hi! I am trying to create a directory within a container, something simple that looks like the code snippet below (in the thread). The task is running successfully but failing to create the data directory. I was thinking that it's supposed to create the directory without any issue, any suggestions to what I am missing please? Thank you
k
Hi @Mukamisha jocelyne, how are you checking if the folder is created? When you get the chance, could you move the code snippet into this thread to keep the main channel cleaner?
m
Copy code
FLOW_NAME = "docker_script"
docker_storage = Docker(
    image_name="test",
    image_tag="latest",
    stored_as_script=True,
    path=f"/app/test3.py",
)


@task(log_stdout=True)
def create_directory():
    text = f"..."
    ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
    local_path = os.path.join(ROOT_DIR, "data")

    Path(local_path).mkdir(parents=True, exist_ok=True)
    return text

with Flow(
    FLOW_NAME,
    storage=docker_storage,
    run_config=DockerRun(
        image="test:latest",
        labels=["docker"],
    ),
) as flow:
    create_dir = create_directory()

if __name__ == "__main__":
    docker_storage.add_flow(flow)
    flow.register(project_name="test_project", build=False)
Sorry about that, I am checking it inside the container created. When running the script without using prefect flow and tasks , I can see the folder being created inside the container but once I use this approach above it doesn't work.
k
No worries I think it would work. I’m more of asking if you are checking using 
docker exec
rather than Docker run?
m
oh okay, I checked using both
k
Docker exec on the new container doesn’t show the new directory? You can this with a Docker agent?
m
No the docker exec doesn't show it
I didn't get the 2nd qsn you asked
k
Sorry typo. You ran* with a Docker agent?
I’ll test this quickly now
m
Oh yes I did run with the docker agent
k
A bit confused. Isn’t the container cleaned up after Flow execution? How do you exec into it?
m
Is the container supposed to clean up? I'm not sure. I opened the terminal on the side and exec into the container produced by the image
k
I think so. What Prefect version are you on?
I am also not sure storing stuff in the Docker container is common practice. We should add a volume maybe and persist data that way. Could you tell me more about the use case here?
m
oh okay, if the container cleans up then that would make sense, i'm using prefect:0.15.12. I was trying create a directory to store some models artifacts pulled down from s3 for generating encodings of images after producing those encodings, I would then store the results on s3.
Thank you, I think I will use volumes for now
k
I think you can just use s3 directly in one flow instead of using a volume then?
46 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by