https://prefect.io logo
Powered by
Title
f

Felix Horvat

08/11/2022, 1:03 PM
Is is possible to have a DockerContainer Block point to a private Container registry - say on gitlab? If so how can I pass the authentication parameters? Or is the assumption that the the Docker service that an agent has access to is logged in to the required container registries?
💯 1
a

Anna Geller

08/11/2022, 2:46 PM
Good question, we started building blocks for login with private registries - so far we only have one for ECR. Would you be open to contribute one for Gitlab following the same Block interface? I can help you along the way, LMK
@Felix Horvat I take that back, I think you can already do that using this block, just set the registry to your Gitlab container registry and in theory this should "just work" - if not, it would be best to open a Github issue, you can cc me there
f

Felix Horvat

08/11/2022, 3:44 PM
@Anna Geller I cannot find the block on my prefect version (cloud 2.0.4). And how would i add the registry to the build command? I can only pass one 
--infra-block
Or would i pass that to the agent? To me it seems that the auth part should just be part of the docker container block - similar to how it is with s3
Can you also point to the existing code for ECR?
a

Anna Geller

08/11/2022, 4:07 PM
It's in the main repo infrastructure/docker - sorry, on mobile now
b

ben

08/23/2022, 6:43 PM
@Felix Horvat Checking whether or not you were able to start anything related to a GitLab Docker Container block contribution. We need to use GitLab too, but can't through the DockerContainer, because we have a server port in our image names. Whatever library is working behind the scenes (docker_py, maybe?) interprets the ":" between the server and port as where the image tag begins. For example, if the image were named "gitlab.server.com:5050/repo/path/shared-image:2.1.1", it will throw an error similar to this. 
docker.errors.APIError: 400 Client Error for <http+docker://localnpipe/v1.41/images/create?tag=5050%2Frepo%2Fpath%2Fshared-image&fromImage=gitlab.server.com>: Bad Request ("invalid tag format")
If no one is working on a contribution yet, I can give it a try as well.
❤️ 1
In 
prefect/infrastructure/docker.py
When I do a proof of concept change (i.e., not production worthy) 
DockerContainer._get_image_and_tag()
and add the noted lines, my GitLab image containing a port number (referenced above) runs successfully: 
def _get_image_and_tag(self) -> Tuple[str, Optional[str]]:
        parts = self.image.split(":")
        if len(parts) > 2:                                      # <= added
            parts = [f"{parts[0]}:{parts[1]}", parts[2]]        # <= added
        image = parts.pop(0)
        tag = parts[0] if parts else None
        return image, tag
With the above adjustment, if I use a generic DockerContainer block together with a generic DockerRegistry block, both connected to a GitLab container registry, both seem to work fine (authenticates successfully, pulls the image, and runs it).
I see that another person has a pull request open for this: https://github.com/PrefectHQ/prefect/pull/6477
a

Anna Geller

08/25/2022, 1:46 PM
thanks for looking into this in detail! I linked this discussion and asked the PR contributor to address the review feedback
b

ben

08/25/2022, 1:53 PM
Funny timing -- I was just coming back here to ask if there is a way that Prefect prefers for me to participate if I have a solution related to an existing PR? Since the PR contributor hadn't followed up for a couple of days I spent a few minutes last night looking at the link that was suggested by madkinsz as a source for proper parsing of a docker image string. I updated the PR author's 
parse_tag_image()
function based on the bounds of possibilities that Docker Image Specification v1.1.0 lists.
I know there are couple of different ways that PRs can be done against an existing PR, but I 100% don't want to step on anyone's toes -- the PR author did the work of figuring out a lot already. I just took what they did and added to it. I mainly just want to be able to use the fix as quickly as I can :) https://github.com/darrida/prefect/blob/cf05964fa1ff2158deffa08e70536fb1a71fa6a0/src/prefect/docker.py#L457
:thank-you: 1
🙌 1
a

Anna Geller

08/25/2022, 2:11 PM
this looks great, could you open this as a separate PR to the Prefect repo, given that the author of the other PR didn't follow up? I think it's fine, we will give credit to you both in release notes (teamwork between contributors!) -- you can just link to the other PR and give the author credit or shoutout
b

ben

08/25/2022, 2:12 PM
That sounds great. I should be able to get PR submitted this morning (it's 9:12am where I am).
:thank-you: 1
💯 1
Created: https://github.com/PrefectHQ/prefect/pull/6564
🙌 1
:thank-you: 1
9 Views
#best-practices-coordination-planeJoin Slack
Powered by