Does anyone have experience running the server on ...
# ask-communitym
Mary Clair Thompson
10/02/2020, 2:11 PMDoes anyone have experience running the server on a machine managed by puppet? I'm running into an issue where seemingly unrelated puppet updates are breaking the back end.
Mary Clair Thompson
10/02/2020, 2:19 PMI'm also looking for documentation on the actual back end? I would like to understand the interaction between the various docker components.
d
Dylan
10/02/2020, 2:22 PM
Hi @Mary Clair Thompson! Here are the Prefect Server docs: https://docs.prefect.io/orchestration/server/overview.html#server-overview
Dylan
10/02/2020, 2:23 PM
The architecture section outlines the interactions between the components
👍 1
m
Mary Clair Thompson
10/02/2020, 2:23 PMWonderful, thank you!
d
Dylan
10/02/2020, 2:23 PM
Of course!
m
Mary Clair Thompson
10/02/2020, 2:23 PMOne problem I'm seeing is that the agent can't seem to talk to the server, even though it appears to be running (all of the Docker containers seem to be running just fine)
d
Dylan
10/02/2020, 2:24 PM
The agent is configured to talk to cloud by default
Dylan
10/02/2020, 2:24 PM
You’ll need to change the API endpoint it’s looking for
Dylan
10/02/2020, 2:25 PM
One moment, I’ll grab the docs
👍 1
Dylan
10/02/2020, 2:25 PM
The CLI command
prefect backend serverDylan
10/02/2020, 2:25 PM
If you were looking to get something working on your machine first
Dylan
10/02/2020, 2:27 PM
m
Mary Clair Thompson
10/02/2020, 2:27 PM@Dylan the agent isn't able to contact the server even after i run prefect backend server
d
Dylan
10/02/2020, 2:27 PM
If you start your agent with
prefect agent start--apiDylan
10/02/2020, 2:28 PM
Are you running this on your machine or is this part of the puppet-controlled setup you were describing previously?
m
Mary Clair Thompson
10/02/2020, 2:29 PMI'm running this on a vm that's being managed by puppet
Mary Clair Thompson
10/02/2020, 2:30 PMIs this the correct syntax? prefect agent start --api localhost:8080
d
Dylan
10/02/2020, 2:31 PM
I believe
localhost:8080Dylan
10/02/2020, 2:31 PM
You’ll need to point at apollo
Dylan
10/02/2020, 2:31 PM
Which is at -checks notes-
m
Mary Clair Thompson
10/02/2020, 2:31 PMhm ok lemme try that
Mary Clair Thompson
10/02/2020, 2:32 PMrequests.exceptions.InvalidSchema: No connection adapters were found for 'localhost:4200'
Mary Clair Thompson
10/02/2020, 2:32 PMthat was the result of running
Mary Clair Thompson
10/02/2020, 2:32 PMprefect agent start --api localhost:4200
d
Dylan
10/02/2020, 2:34 PM
That sounds like the db migrations weren’t run by postgres/hasura
Dylan
10/02/2020, 2:35 PM
The single-node deployment of Prefect Server is controlled by the CLI command
prefect server startm
Mary Clair Thompson
10/02/2020, 2:35 PMright right
d
Dylan
10/02/2020, 2:35 PM
I’m looking for the script
Dylan
10/02/2020, 2:35 PM
Because if you’re trying to roll a custom deployment you’re going to need to do all of the things in that script
m
Mary Clair Thompson
10/02/2020, 2:37 PMI'm not entirely sure what you mean by 'custom deployment'...are you referring to the puppet bits?
d
Dylan
10/02/2020, 2:37 PM
Sorry, I assumed that you’re looking to run each service on a different machine managed by puppet?
m
Mary Clair Thompson
10/02/2020, 2:38 PMnope
d
Dylan
10/02/2020, 2:38 PM
Are you trying to run server on a single node?
m
Mary Clair Thompson
10/02/2020, 2:38 PMyes
Mary Clair Thompson
10/02/2020, 2:38 PMwhole shebang
d
Dylan
10/02/2020, 2:38 PM
Ah okay
Dylan
10/02/2020, 2:38 PM
prefect server startDylan
10/02/2020, 2:38 PM
hmm
Dylan
10/02/2020, 2:38 PM
What version of Prefect are you on?
m
Mary Clair Thompson
10/02/2020, 2:38 PMlatest
Mary Clair Thompson
10/02/2020, 2:38 PM0.13.9
Mary Clair Thompson
10/02/2020, 2:39 PMLet me walk through mysetup
d
Dylan
10/02/2020, 2:39 PM
That would be helpful 😄
m
Mary Clair Thompson
10/02/2020, 2:39 PMI installed everything on a single vm.
d
Dylan
10/02/2020, 2:39 PM
Also, do you see any errors when running
prefect server startm
Mary Clair Thompson
10/02/2020, 2:39 PMnot initially
Mary Clair Thompson
10/02/2020, 2:40 PMLet me run through this and i'll explain what i'm seeing as I go along
Mary Clair Thompson
10/02/2020, 2:40 PMso let's not think about puppet for now, i'm not sure what kind of schedule it's running on as i don't manage it
👍 1
Mary Clair Thompson
10/02/2020, 2:41 PMyesterday I installed prefect on this machine. started the server and a local agent. everything looked great. I was able to open a port from my local machine to the vm and navigate to the ui and see a flow i had registered
Mary Clair Thompson
10/02/2020, 2:42 PMTHEN puppet runs. At this point all of the docker containers appear to be up and running, but when i try to navigate to the UI, i get basically just the top bar of the UI and nothing else
Mary Clair Thompson
10/02/2020, 2:42 PMin addition, starting a new agent fails
Mary Clair Thompson
10/02/2020, 2:42 PMand this is even though the server appears to be up
Mary Clair Thompson
10/02/2020, 2:42 PMjust for kicks, i tried running
Mary Clair Thompson
10/02/2020, 2:42 PMprefect server start
Mary Clair Thompson
10/02/2020, 2:43 PMagain, and yes i got lots of errors
Mary Clair Thompson
10/02/2020, 2:45 PMIf i restart the vm, everything runs fine again, until puppet does its thing
Mary Clair Thompson
10/02/2020, 2:45 PMso it looks like puppet is messing with the containers in some way
Mary Clair Thompson
10/02/2020, 2:47 PMAh ok interesting--i just tried starting the server again and it ran fine. So i think when puppet runs it's killing the server somehow
d
Dylan
10/02/2020, 2:49 PM
That sounds like a promising lead!
Dylan
10/02/2020, 2:49 PM
I am totally unfamiliar with puppet but I’m definitely interested to hear what you find out
👍 1
m
Mary Clair Thompson
10/02/2020, 2:50 PMThanks for your help Dylan! hoping we can take it from here...
d
Dylan
10/02/2020, 2:50 PM
Best of luck
m
Mary Clair Thompson
10/02/2020, 5:09 PMWe found the issue--puppet was closing a bunch of firewall holes on restart. Thanks again for your help @Dylan!
d
Dylan
10/02/2020, 5:09 PM
I’m glad you got it figured out!