Hello, I have an issue with the storage, as it's getting hard to configure the TLS config with DockerStorage as our registry is not properly administrated yet (another part of my team is handling this) a few questions: • Is there a way to use Docker storage without pushing the image to a repository ? • Can't we authentify to docker repository with login/password instead of TLS Config ? • Is there a way to specifiy the files to upload when using S3/GCS ? I first tried theses options but I got many file not found errors due to only the flow file being uploaded. I switch to docker storage because I saw it had the 'files' option.

Hi @Emma Rizzi, yes it is possible to use Docker storage without pushing to a registry if the flow will be run in a place that already has that image and using the Docker a agent. How are you currently authenticating? I think authenticating might be a separate operation from Prefect, and it would be a matter of giving your agent access to that registry.

@Kevin Kho No I use a ECS Agent running on another instance, I suspected it would be the problem anyway I usually authenticate with docker login (username + pwd), but I see on the DockerStorage documentation that only TLS Config is possible I didn't find any discussion on how to authenticate separatly from Prefect, thats would be a good solution for me if its possible I see Docker is the best option as I rely on local files as much as python modules, I hope I can make this work !

Are you using ECR for you registry or something else?

Not ECR, its a sonatype nexus registry, we would like to keep like this to keep control over the backend storage if its possible

I’ll look into this a bit more and get back to you

I think I misunderstood many things here as I'm not very familiar with docker yet, but I get it know when you said about authenticate separatly from Prefect ! I found the AWS doc to authenticate my instances (here the link if anyone gets stuck on this too) I'll try this and hopefully it will be all I need 🙂

Yeah that’s what I meant. I need to figure out myself how it works with Prefect 🙂

Hi again fellow Groot lover, I looked a bit more into this. I did

docker login

on the instance hosting Prefect, the pull worked, I'm almost there! Now the pull seems more problematic, according to AWS doc I sent, it requires only 2 env vars to pass "when starting the agent". I followed the instruction to ssh to the instance, add vars to ecs.config, I tried to add the vars to RunCongif env vars, and also to the agent when starting it (with --env option) Each time I got this error on AWS console :

CannotPullContainerError: inspect image has been retried 1 time(s): failed to resolve ref "<http://nexus-release.XXX/multitask_container:2021-06-16t08-00-50-057792-00-00|nexus-release.XXX/multitask_container:2021-06-16t08-00-50-057792-00-00>": unexpected status code [manifests 2021-06-16t08-00-50-057792-00-00]:...

As I can pull the image locally, I guess the problem comes from the authentification. I tried looking into Prefect code to see how the tasks are launched. Another doc : https://docs.aws.amazon.com/AmazonECS/latest/developerguide/private-auth.html says to give private registry logins to the task definition I see in the code here that the default container definition is created with only "image", while boto doc indicates we can also provide private registry credentials. I believe this is the only thing missing to run tasks with private images

I just got on...you got it? Thanks for the info here!

Yes the final solution seems rather simple actually! to sum up I only needed to

docker login

on the instance hosting Prefect and give a task_definition following this doc to my ECS Agent 🙂

Nice! Glad you figured it out!

Sure! I've never wrote anything before so I'd love to have a first article, I'll try to free some time for this as it could be helpfull for others 🙂