https://prefect.io logo
Channels
Powered by
m

Mary Clair Thompson

10/02/2020, 2:11 PM
Does anyone have experience running the server on a machine managed by puppet? I'm running into an issue where seemingly unrelated puppet updates are breaking the back end.
I'm also looking for documentation on the actual back end? I would like to understand the interaction between the various docker components.
d

Dylan

10/02/2020, 2:22 PM
Hi @Mary Clair Thompson! Here are the Prefect Server docs: https://docs.prefect.io/orchestration/server/overview.html#server-overview
The architecture section outlines the interactions between the components
👍 1
m

Mary Clair Thompson

10/02/2020, 2:23 PM
Wonderful, thank you!
d

Dylan

10/02/2020, 2:23 PM
Of course!
m

Mary Clair Thompson

10/02/2020, 2:23 PM
One problem I'm seeing is that the agent can't seem to talk to the server, even though it appears to be running (all of the Docker containers seem to be running just fine)
d

Dylan

10/02/2020, 2:24 PM
The agent is configured to talk to cloud by default
You’ll need to change the API endpoint it’s looking for
One moment, I’ll grab the docs
👍 1
The CLI command 
prefect backend server
sets a config file that tells the agent to talk to the default server endpoint for a single-node deployment
If you were looking to get something working on your machine first
m

Mary Clair Thompson

10/02/2020, 2:27 PM
@Dylan the agent isn't able to contact the server even after i run prefect backend server
d

Dylan

10/02/2020, 2:27 PM
If you start your agent with 
prefect agent start
you can pass it an api endpoint with 
--api
Are you running this on your machine or is this part of the puppet-controlled setup you were describing previously?
m

Mary Clair Thompson

10/02/2020, 2:29 PM
I'm running this on a vm that's being managed by puppet
Is this the correct syntax? prefect agent start --api localhost:8080
d

Dylan

10/02/2020, 2:31 PM
I believe 
localhost:8080
is the UI endpoint
You’ll need to point at apollo
Which is at -checks notes-
m

Mary Clair Thompson

10/02/2020, 2:31 PM
hm ok lemme try that
requests.exceptions.InvalidSchema: No connection adapters were found for 'localhost:4200'
that was the result of running
prefect agent start --api localhost:4200
d

Dylan

10/02/2020, 2:34 PM
That sounds like the db migrations weren’t run by postgres/hasura
The single-node deployment of Prefect Server is controlled by the CLI command 
prefect server start
m

Mary Clair Thompson

10/02/2020, 2:35 PM
right right
d

Dylan

10/02/2020, 2:35 PM
I’m looking for the script
Because if you’re trying to roll a custom deployment you’re going to need to do all of the things in that script
m

Mary Clair Thompson

10/02/2020, 2:37 PM
I'm not entirely sure what you mean by 'custom deployment'...are you referring to the puppet bits?
d

Dylan

10/02/2020, 2:37 PM
Sorry, I assumed that you’re looking to run each service on a different machine managed by puppet?
m

Mary Clair Thompson

10/02/2020, 2:38 PM
nope
d

Dylan

10/02/2020, 2:38 PM
Are you trying to run server on a single node?
m

Mary Clair Thompson

10/02/2020, 2:38 PM
yes
whole shebang
d

Dylan

10/02/2020, 2:38 PM
Ah okay
prefect server start
should run the migrations and whatnot then
hmm
What version of Prefect are you on?
m

Mary Clair Thompson

10/02/2020, 2:38 PM
latest
0.13.9
Let me walk through mysetup
d

Dylan

10/02/2020, 2:39 PM
That would be helpful 😄
m

Mary Clair Thompson

10/02/2020, 2:39 PM
I installed everything on a single vm.
d

Dylan

10/02/2020, 2:39 PM
Also, do you see any errors when running 
prefect server start
?
m

Mary Clair Thompson

10/02/2020, 2:39 PM
not initially
Let me run through this and i'll explain what i'm seeing as I go along
so let's not think about puppet for now, i'm not sure what kind of schedule it's running on as i don't manage it
👍 1
yesterday I installed prefect on this machine. started the server and a local agent. everything looked great. I was able to open a port from my local machine to the vm and navigate to the ui and see a flow i had registered
THEN puppet runs. At this point all of the docker containers appear to be up and running, but when i try to navigate to the UI, i get basically just the top bar of the UI and nothing else
in addition, starting a new agent fails
and this is even though the server appears to be up
just for kicks, i tried running
prefect server start
again, and yes i got lots of errors
If i restart the vm, everything runs fine again, until puppet does its thing
so it looks like puppet is messing with the containers in some way
Ah ok interesting--i just tried starting the server again and it ran fine. So i think when puppet runs it's killing the server somehow
d

Dylan

10/02/2020, 2:49 PM
That sounds like a promising lead!
I am totally unfamiliar with puppet but I’m definitely interested to hear what you find out
👍 1
m

Mary Clair Thompson

10/02/2020, 2:50 PM
Thanks for your help Dylan! hoping we can take it from here...
d

Dylan

10/02/2020, 2:50 PM
Best of luck
m

Mary Clair Thompson

10/02/2020, 5:09 PM
We found the issue--puppet was closing a bunch of firewall holes on restart. Thanks again for your help @Dylan!
d

Dylan

10/02/2020, 5:09 PM
I’m glad you got it figured out!
Powered by