Zach Schumacher

    Zach Schumacher

    1 year ago
    Hey guys! Have a kubernetes run question. I have a use case where I need to dynamically set a kubernetes run config. I want to find out what the latest tag is of a docker image, and then use that same docker image for each task in a flow. I don’t know what the latest tag is at registration time, only at run time. Using :latest does not work, because it is not impossible :latest could change between the first task in a flow and the last task in a flow. Is there a recipe for doing this?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Zach Schumacher, does the docker image live in a container registry or on your local?
    Zach Schumacher

    Zach Schumacher

    1 year ago
    it is in docker hub
    Kevin Kho

    Kevin Kho

    1 year ago
    My best idea is to scrape the tag from the dockerhub page. I don’t think their API supports this operation.
    You can go to this equivalent page and get the top entry?
    Zach Schumacher

    Zach Schumacher

    1 year ago
    no thats not my problem. The problem is i need to dynamically register the run config of the flow. I can get the latest tag from github no problem.
    Kevin Kho

    Kevin Kho

    1 year ago
    I know what you mean. I’ll look into this.
    Zach Schumacher

    Zach Schumacher

    1 year ago
    the answer can be “you need to figure that out with k8s and its not a prefect problem,” but just wanted to see if anyone else had that requirement before
    Kevin Kho

    Kevin Kho

    1 year ago
    I am a bit confused why something like the code snippet here doesn’t work? get_tag is a function here, not a task.
    def get_tag():
        # do something
    with Flow("test") as flow:
        # do something
    
    flow.run()
    flow.run_config = KubernetesRun(image=get_tag())
    Zach Schumacher

    Zach Schumacher

    1 year ago
    doesn’t that run when a flow gets registered
    or will that get refreshed each time?
    Kevin Kho

    Kevin Kho

    1 year ago
    Tasks don’t get run but Python functions will.
    Zach Schumacher

    Zach Schumacher

    1 year ago
    right, so i want the get_tag code to be ran at flow run time
    not sure if thats possible since the image gets set when a flow is registered
    Kevin Kho

    Kevin Kho

    1 year ago
    I see. That is not possible but the best advice I can give you then is using the Docker Sidecar and managing the containers this way.
    You would have to run the container through a task instead of having a task be run on the container.
    Zach Schumacher

    Zach Schumacher

    1 year ago
    i think this is exactly what i want, thanks!
    Kevin Kho

    Kevin Kho

    1 year ago
    Sorry for the detours 😅. Hope you get this working.