Thread
#prefect-community
    a

    Alex Welch

    1 year ago
    hello, I am running into an issue with an
    ECS Agent
    . I get the below error on my flow run. I have checked, and found, the
    3.8.5
    version (it was the docker container I built the flow in), but I can not determine where the
    3.7.10
    version is. I looked in the
    EC2
    instance that I set up and launched the agent from and it is at
    3.8.7
    . Where else is Prefect running the flow that may have the
    3.7
    Python version and is there any way around this other than upgrading that to
    3.8
    ?
    Looks like it was in the
    prefecthq/prefect:latest
    image
    Chris White

    Chris White

    1 year ago
    Makes sense — we have multiple images for different python versions that you can check out here: https://hub.docker.com/r/prefecthq/prefect
    a

    Alex Welch

    1 year ago
    sweet
    i do have one other question i just ran into
    Chris White

    Chris White

    1 year ago
    Fire away
    a

    Alex Welch

    1 year ago
    i have a docker image in a private repo, and prefect can’t get to it because its unauthorized. IS there someway I can pass authorization in the flow configs, or do I need to set up ssh keys on the ECS agent
    its a Github repo if that matters at all
    my runconfig loks like
    RUN_CONFIG = ECSRun(labels=['s3-flow-storage'],
                          task_role_arn=TASK_ARN,
                          image=PREFECT_DATA_IMAGE,
                          memory=512,
                          cpu=256
                        )
    Chris White

    Chris White

    1 year ago
    Hmmm at the moment that needs to be set up on the ECS agent; the reason this isn’t available as a run config setting is essentially because the different agents all have drastically different auth patterns (e.g., docker agents use the docker daemon config, k8s agents use cluster-level config, etc.)
    and all require some amount of configuration external to prefect
    a

    Alex Welch

    1 year ago
    yeah i figured, just wanted to make sure i was on the right path
    i seem to be climbing over one hurdle and running into the next one
    i always feel like i am sooooo close to getting it fully working
    lol you guys dont offer consulting to help get it set up do you
    Chris White

    Chris White

    1 year ago
    haha yea, we’re very aware of the DevOps requirements for getting a production ready setup and are actively brainstorming ideas for how to streamline the whole process. We actually do have an amazing partner that offers consulting services for Prefect! If you’re interested in learning more feel free to shoot me an email chris at prefect.io
    a

    Alex Welch

    1 year ago
    i will do that because this is the last piece of my puzzle
    Chris White

    Chris White

    1 year ago
    awesome — and no pressure, but if you document some of the hurdles you experienced we will absolutely take them into consideration for documentation and feature enhancements
    a

    Alex Welch

    1 year ago
    i started to
    alright, just shot it over
    Chris White

    Chris White

    1 year ago
    received - i’ll get you connected with the right folks 👍
    a

    Alex Welch

    1 year ago
    thanks
    thinking out loud as i do this, it would be pretty baller if you guys had example terraform templates for setting up different environments
    Chris White

    Chris White

    1 year ago
    In all seriousness we were discussing some feedback today requesting helm charts — I’ll link this to the team as another thing to consider 👍
    a

    Alex Welch

    1 year ago
    very cool
    so there isnt anything analogous to Airflow’s
    image_pull_secrets
    ? I just set up both SSH and docker login on the EC2 agent machine, but im still getting
    Unauthorized
    . I think it is because the agent is spinning up another resource and not passing the credentials
    looks like it would go in the
    containerDefinitions
    json being passed to the task
    would that be a custom task definition? and would that conflict with the ‘image’ key already being passed?
    Chris White

    Chris White

    1 year ago
    The kubernetes run config does have an image pull secret but it doesn’t appear the ECS config does; I’m not very fluent in AWS, I think for a higher quality response on the image pull secrets aspect you might want to open a GitHub issue (or start a new thread for others to chime in on) — I’m confident this is a popular requirement for folks
    a

    Alex Welch

    1 year ago
    but would that be an example of what the
    custom task definition
    argument is for?
    Chris White

    Chris White

    1 year ago
    yup exactly