Thread
#prefect-community
    Darragh

    Darragh

    2 years ago
    Back with another bunch of stupid! Today’s questions… • I build my flow into a Docker storage, and register it with the Prefect Server I have running on AWS - what’s the simplest execution model I can use to get it running? I’m assuming if I’ve built it locally then the registry I push it to needs to be accessible by both the builder [local machine] and the server? • Am I right in assuming I want a RemoteEnvironment with LocalExecutor? And does the execution agent need to be a Docker agent? • The agent running alongside my server is crashing out because it’s trying to connect to http://localhost:4200 rather than the IP of the AWS EC2 instance it’s running on - is there an environment variable override for this?
    [2020-05-18 17:52:01,285] ERROR - agent | HTTPConnectionPool(host=‘localhost’, port=4200): Max retries exceeded with url: /graphql/alpha (Caused by NewConnectionError(‘<urllib3.connection.HTTPConnection object at 0x7f52470ae0f0>: Failed to establish a new connection: [Errno 111] Connection refused’,))
    Thanks in advance 🙂
    Tyler Wanner

    Tyler Wanner

    2 years ago
    lol these are not stupid questions! These answers probably belong in a followup blog post
    I would say yes yes and yes 😃
    PREFECT__CLOUD__API should do the trick
    Darragh

    Darragh

    2 years ago
    Thanks Tyler, working though those pieces now - PREFECT__CLOUD__API , should that be set to the IP, or full path to graphql?
    Latest error, assuming it’s credentials based:
    Error while deploying flow: APIError(HTTPError('500 Server Error: Internal Server Error for url: <http+docker://localhost/v1.40/images/create?tag=latest&fromImage=>...........
    A lot of it is in various docs but it’s piecing it all together for my specifica case I’m finding difficult 🙂 For instance, since my docker registry is a private one on amazon, what way do I inject the creds to the [server/flow/agent] so it can pull the image and run??
    Tyler Wanner

    Tyler Wanner

    2 years ago
    building a distributed workflow orchestration system that runs reliably in the cloud and properly authenticates with everything is hard, and requires spending time away from that which you’re actually trying to accomplish--that’s why we built Prefect Cloud 😄
    I believe the IP should be enough for the env var
    Let me get back to you on the others 🙂
    Darragh

    Darragh

    2 years ago
    Nice try on the cloud push there 😂
    nicholas

    nicholas

    2 years ago
    Hi again @Darragh! You're right that you'll need to authenticate whatever machine your agent is running on with your image registry (it looks like ECR). I'm not sure the specific steps for authenticating with ECR but it should match what you've done locally to reach the repository
    Darragh

    Darragh

    2 years ago
    Hmm, but do i not need to pass creds to the docker agent? Or is it that the command to start the agent can take in params? Im still hazy on the general secrets model for this..
    nicholas

    nicholas

    2 years ago
    No it's not the agent that'll need creds, the Docker daemon itself needs to be authenticated as it would be locally to talk to a private registry
    So no secrets (as far as Prefect is concerned) need to be involved
    Darragh

    Darragh

    2 years ago
    Interesting... So on the ec2 instance im running everything on, before i start the prefect server and the agent i just login to my docker registry and the pull should work once the agent starts up?
    nicholas

    nicholas

    2 years ago
    You should only need to do it before you start the agent 🙂
    Darragh

    Darragh

    2 years ago
    Magnifique, worked like a charm!
    nicholas

    nicholas

    2 years ago
    Woohoo! Good job 😄
    Darragh

    Darragh

    2 years ago
    Thanks for all the advice! Need to figure out a long term strategy as I don’t know if the login method I’m using will allow for long lived logins, but progress is progress, and I’ve finally managed to get a basic flow running all the way through my batshit conatinerized CI/CD & AWS-CDK setup 😂
    Jacob Blanco

    Jacob Blanco

    2 years ago
    We are in a similar boat her trying to setup the Agent in an EC2 instance pulling from ECR. We are testing right now. We are having issue combining running the agent in a way where we can disconnected from the server and having it pick up the new Docker session after a token refresh.