Emma Rizzi

    Emma Rizzi

    1 year ago
    Hello, I have an issue with the storage, as it's getting hard to configure the TLS config with DockerStorage as our registry is not properly administrated yet (another part of my team is handling this) a few questions: • Is there a way to use Docker storage without pushing the image to a repository ? • Can't we authentify to docker repository with login/password instead of TLS Config ? • Is there a way to specifiy the files to upload when using S3/GCS ? I first tried theses options but I got many file not found errors due to only the flow file being uploaded. I switch to docker storage because I saw it had the 'files' option.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Emma Rizzi, yes it is possible to use Docker storage without pushing to a registry if the flow will be run in a place that already has that image and using the Docker a agent. How are you currently authenticating? I think authenticating might be a separate operation from Prefect, and it would be a matter of giving your agent access to that registry.
    I think for the last question, if you have other dependencies like custom Python modules, you have to use Docker instead of S3/GCS. We recommend people install the library on the image.
    Emma Rizzi

    Emma Rizzi

    1 year ago
    @Kevin Kho No I use a ECS Agent running on another instance, I suspected it would be the problem anyway I usually authenticate with docker login (username + pwd), but I see on the DockerStorage documentation that only TLS Config is possible I didn't find any discussion on how to authenticate separatly from Prefect, thats would be a good solution for me if its possible I see Docker is the best option as I rely on local files as much as python modules, I hope I can make this work !
    Do you have any information on how to give access to the registry tp my ECS agent ? I'll try to look more into this option
    Kevin Kho

    Kevin Kho

    1 year ago
    Are you using ECR for you registry or something else?
    Emma Rizzi

    Emma Rizzi

    1 year ago
    Not ECR, its a sonatype nexus registry, we would like to keep like this to keep control over the backend storage if its possible
    Kevin Kho

    Kevin Kho

    1 year ago
    I’ll look into this a bit more and get back to you
    Emma Rizzi

    Emma Rizzi

    1 year ago
    I think I misunderstood many things here as I'm not very familiar with docker yet, but I get it know when you said about authenticate separatly from Prefect ! I found the AWS doc to authenticate my instances (here the link if anyone gets stuck on this too) I'll try this and hopefully it will be all I need 🙂
    Kevin Kho

    Kevin Kho

    1 year ago
    Yeah that’s what I meant. I need to figure out myself how it works with Prefect 🙂
    Emma Rizzi

    Emma Rizzi

    1 year ago
    Hi again fellow Groot lover, I looked a bit more into this. I did
    docker login
    on the instance hosting Prefect, the pull worked, I'm almost there! Now the pull seems more problematic, according to AWS doc I sent, it requires only 2 env vars to pass "when starting the agent". I followed the instruction to ssh to the instance, add vars to ecs.config, I tried to add the vars to RunCongif env vars, and also to the agent when starting it (with --env option) Each time I got this error on AWS console :
    CannotPullContainerError: inspect image has been retried 1 time(s): failed to resolve ref "<http://nexus-release.XXX/multitask_container:2021-06-16t08-00-50-057792-00-00|nexus-release.XXX/multitask_container:2021-06-16t08-00-50-057792-00-00>": unexpected status code [manifests 2021-06-16t08-00-50-057792-00-00]:...
    As I can pull the image locally, I guess the problem comes from the authentification. I tried looking into Prefect code to see how the tasks are launched. Another doc : https://docs.aws.amazon.com/AmazonECS/latest/developerguide/private-auth.html says to give private registry logins to the task definition I see in the code here that the default container definition is created with only "image", while boto doc indicates we can also provide private registry credentials. I believe this is the only thing missing to run tasks with private images
    Me again, I finally made it worked by writting the task_definition myself. I'll see with my team if I could contribute with an article about all this as I haven't found any in my particular set up it could be usefull. Thanks a lot for pointing me to the right direction @Kevin Kho
    Kevin Kho

    Kevin Kho

    1 year ago
    I just got on...you got it? Thanks for the info here!
    Emma Rizzi

    Emma Rizzi

    1 year ago
    Yes the final solution seems rather simple actually! to sum up I only needed to
    docker login
    on the instance hosting Prefect and give a task_definition following this doc to my ECS Agent 🙂
    Kevin Kho

    Kevin Kho

    1 year ago
    Nice! Glad you figured it out!
    If you make a blog btw, we’d be happy to feature it on our site also
    Emma Rizzi

    Emma Rizzi

    1 year ago
    Sure! I've never wrote anything before so I'd love to have a first article, I'll try to free some time for this as it could be helpfull for others 🙂