https://prefect.io logo
d

Darragh

05/20/2020, 3:43 PM
Back with another bunch of stupid! Today’s questions… • I build my flow into a Docker storage, and register it with the Prefect Server I have running on AWS - what’s the simplest execution model I can use to get it running? I’m assuming if I’ve built it locally then the registry I push it to needs to be accessible by both the builder [local machine] and the server? • Am I right in assuming I want a RemoteEnvironment with LocalExecutor? And does the execution agent need to be a Docker agent? • The agent running alongside my server is crashing out because it’s trying to connect to http://localhost:4200 rather than the IP of the AWS EC2 instance it’s running on - is there an environment variable override for this?
[2020-05-18 17:52:01,285] ERROR - agent | HTTPConnectionPool(host=‘localhost’, port=4200): Max retries exceeded with url: /graphql/alpha (Caused by NewConnectionError(‘<urllib3.connection.HTTPConnection object at 0x7f52470ae0f0>: Failed to establish a new connection: [Errno 111] Connection refused’,))
Thanks in advance 🙂
t

Tyler Wanner

05/20/2020, 4:07 PM
lol these are not stupid questions! These answers probably belong in a followup blog post
I would say yes yes and yes 😃
PREFECT__CLOUD__API should do the trick
d

Darragh

05/20/2020, 4:34 PM
Thanks Tyler, working though those pieces now - PREFECT__CLOUD__API , should that be set to the IP, or full path to graphql?
Latest error, assuming it’s credentials based:
Error while deploying flow: APIError(HTTPError('500 Server Error: Internal Server Error for url: <http+docker://localhost/v1.40/images/create?tag=latest&fromImage=>...........
A lot of it is in various docs but it’s piecing it all together for my specifica case I’m finding difficult 🙂 For instance, since my docker registry is a private one on amazon, what way do I inject the creds to the [server/flow/agent] so it can pull the image and run??
t

Tyler Wanner

05/20/2020, 6:00 PM
building a distributed workflow orchestration system that runs reliably in the cloud and properly authenticates with everything is hard, and requires spending time away from that which you’re actually trying to accomplish--that’s why we built Prefect Cloud 😄
upvote 2
I believe the IP should be enough for the env var
Let me get back to you on the others 🙂
d

Darragh

05/20/2020, 6:16 PM
Nice try on the cloud push there 😂
n

nicholas

05/20/2020, 6:34 PM
Hi again @Darragh! You're right that you'll need to authenticate whatever machine your agent is running on with your image registry (it looks like ECR). I'm not sure the specific steps for authenticating with ECR but it should match what you've done locally to reach the repository
d

Darragh

05/20/2020, 6:43 PM
Hmm, but do i not need to pass creds to the docker agent? Or is it that the command to start the agent can take in params? Im still hazy on the general secrets model for this..
n

nicholas

05/20/2020, 6:59 PM
No it's not the agent that'll need creds, the Docker daemon itself needs to be authenticated as it would be locally to talk to a private registry
👍 1
So no secrets (as far as Prefect is concerned) need to be involved
d

Darragh

05/20/2020, 7:31 PM
Interesting... So on the ec2 instance im running everything on, before i start the prefect server and the agent i just login to my docker registry and the pull should work once the agent starts up?
👍 2
n

nicholas

05/20/2020, 7:32 PM
You should only need to do it before you start the agent 🙂
d

Darragh

05/20/2020, 7:58 PM
Magnifique, worked like a charm!
n

nicholas

05/20/2020, 7:59 PM
Woohoo! Good job 😄
d

Darragh

05/20/2020, 8:02 PM
Thanks for all the advice! Need to figure out a long term strategy as I don’t know if the login method I’m using will allow for long lived logins, but progress is progress, and I’ve finally managed to get a basic flow running all the way through my batshit conatinerized CI/CD & AWS-CDK setup 😂
😆 1
🙏 1
j

Jacob Blanco

05/29/2020, 6:42 AM
We are in a similar boat her trying to setup the Agent in an EC2 instance pulling from ECR. We are testing right now. We are having issue combining running the agent in a way where we can disconnected from the server and having it pick up the new Docker session after a token refresh.
2 Views