https://prefect.io logo
j

Jeff Williams

01/07/2021, 7:39 PM
Hi all. I hope I am in the right channel. I am trying to set up Prefect in a GCP account using a micro-VM as discussed in this article: https://medium.com/the-prefect-blog/prefect-server-101-deploying-to-google-cloud-platform-47354b16afe2. What is happening is that I an getting time out errors on the spin up of tmp_apollo_1 and tmp_towel_1 that say "(UnixHTTPConnectionPool (host='localhost', port=None): Read timed out. (read timeout=60)". Then there is some reference to a possible slow network. Everything else up to that point has worked fine, per the article. When I ping my localhost from the ssh window, it responds immediately, so I don't think it is a slow network issue. I could possibly be a configuration issue as the environment I am working in has to be tightly controlled due to compliance issues. But I think I have the appropriate Firewall Rules in place to allow the right things. Again, using the article as my road map. Any help would be greatly appreciated.
z

Zanie

01/07/2021, 7:52 PM
j

Jeff Williams

01/07/2021, 8:14 PM
Thanks. I have read through the thread and will see if my issue gets resolved.
I was able to get the server to start up by changing the VM that I was using. I can at least get the banner to show up now.
z

Zanie

01/08/2021, 3:41 PM
Interesting. I’ll let the author of the article know.
j

Jeff Williams

01/08/2021, 3:43 PM
I am going to try going back to the same size VM as mentioned in the article, but use some of the other changes that I made when I moved to the larger VM. This is still a POC for me and I will be happy to share the results later today.
z

Zanie

01/08/2021, 3:43 PM
Great! Thanks
n

nicholas

01/08/2021, 3:51 PM
Hi @Jeff Williams - sorry for the confusion on that, I realize there's a screenshot of the instance creation view in GCP, but that wasn't intended to be an instruction on how large/small a machine to use. Prefect Server currently consists of 7 nodes, each running as containers through Docker, and so is a nontrivial application stack. Because of this, you'll need to use a larger VM type than
micro
and probably even larger still if you plan to scale out your workloads (and don't forget to attach a volume!)
j

Jeff Williams

01/08/2021, 4:08 PM
Good to know @nicholas. The screenshot did lead me to believe that was a pseudo-recommendation. My bad...
n

nicholas

01/08/2021, 4:13 PM
I'd updated the description on that picture to avoid confusion; thanks for bringing that to our attention 🙂
j

Jeff Williams

01/08/2021, 4:14 PM
I will make a point to review the Architecture link you provided as well... 😉
😄 1
@nicholas - After verifying certain items with our internal network for this install, I am still having trouble getting the UI to show up. Looking at the output when the server starts up, it "almost" seems like it isn't using the config.toml file that is in ~./prefect directory as directed in the article. I have made the necessary edits to ensure it is set for our IP address, but I keep seeing the statement "Server ready at http://0.0.0.0:4200". The PostgreSQL server output also references 0.0.0.0 at various points. Any ideas?
n

nicholas

01/15/2021, 6:13 PM
Hi @Jeff Williams - when you say you're having trouble getting the UI to show up, do you mean that you can't access the UI at all or that it isn't connecting to the Apollo server correctly?
j

Jeff Williams

01/15/2021, 6:18 PM
When I try to use my browser window to get to the UI, as directed, I does not connect. Working within the confines of our firewall rules, I have opened it up to allow any IP address from within our organization using any protocol to any port to connect to the GCP VM. That is when I started looking at the on-screen output and noticing references to the server being at 0.0.0.0 instead of the IP address assigned in the config.toml file.
I see a few messages from the apollo server stating that it is "Checking GraphQL service at http://graphql:4201/health ..."
Then later I see it saying "GraphQL service Health!", some other info, and then "Building schema..." Then complete, then the Server ready statement I listed above. All from the apollo_1 output.
@nicholas - Good morning. Did you have any insight based on my comments?
n

nicholas

01/19/2021, 3:50 PM
Hi @Jeff Williams - unfortunately I don't. If you can't get to the UI, that tells me your local machine can't see the address over which you're serving it. Given you've got some internal firewall rules that could be preventing you from accessing your deployment, I'm not sure how much I can diagnose on this end.
j

Jeff Williams

01/19/2021, 4:28 PM
@nicholas - Understandable. Once I get it figured out, if it is relevant, I will share with you the solution in case it is helpful for someone else.
👍 1
n

nicholas

01/19/2021, 4:35 PM
Much appreciated, if you have any questions about any of the Server internals that might be helpful lmk!
Hi @Jeff Williams - did you figure out your issue? I saw your message but didn't have a chance to respond
j

Jeff Williams

01/20/2021, 5:02 PM
@nicholas - Yes, and it was user error evidently so I deleted the evidence.... 🤣🤣🤣🤣 I am going to try and move forward now....
n

nicholas

01/20/2021, 5:07 PM
Haha no shame in that! Happens to all of us 🙂
j

Jeff Williams

01/27/2021, 8:26 PM
@nicholas - I have been working on a few different solutions and what I have settled on is to run the server with the --no-gui option so that all other components will load up & start. Then, I have downloaded the UI git repo and built a "custom" image where I am adding a self-signed cert (for now) to allow the UI to run on port 443 (SSL - A requirement for my industry) I have also made the change in the nginx.conf to listen on 443 rather than 8080. It SEEMS to be working, in that the webpage will load and change the title on the tab to "Prefect". However, it is not showing any data. I simply get a blank screen. When I issue the "docker container logs <container>" command, I am getting log data but nothing tells me one way or the other if everything is talking or not talking to each other. How can I tell? Do you have any troubleshooting tips for me?
n

nicholas

01/27/2021, 8:27 PM
Hi @Jeff Williams - can you provide a screenshot of what you're seeing, including the javascript console?
j

Jeff Williams

01/27/2021, 8:29 PM
@nicholas - Do you mean an F12 console "DOM Explorer" screenshot or something from the docker logs?
n

nicholas

01/27/2021, 8:30 PM
If you press F12 (or cmd+option+i on mac) and then navigate to the
Console
tab, you'll see a javascript developer console 🙂
j

Jeff Williams

01/27/2021, 8:31 PM
Actually, I see a message under the <noscript> tag saying that JavaScript is not enabled. Let me see if that is my issue and I will get back with you...
n

nicholas

01/27/2021, 8:32 PM
Hm, that's a standard tag on most SPAs, if it's not displaying in the browser then you should be fine
j

Jeff Williams

01/27/2021, 8:43 PM
Well, if that is the case, my F12 doesn't show a JavaScript developer console. I see options for: "DOM Explorer", Console, Debugger, Network, Performance, Memory, and Emulation.
n

nicholas

01/27/2021, 8:43 PM
Right,
Console
is what you're looking for
j

Jeff Williams

01/27/2021, 8:46 PM
All it lists is 3 - "i" icons with "Current window" information.
n

nicholas

01/27/2021, 8:47 PM
Could you screenshot what you're seeing?
j

Jeff Williams

01/27/2021, 8:51 PM
Yes. It is on a different system that I can't load Slack onto so I have to do some hoop jumping.... 🙂
Here it is:
n

nicholas

01/27/2021, 8:55 PM
Hm ok, there are a few issues I'm seeing: 1) you've got a certificate error, which means most browsers won't serve anything and 2) we only support modern browsers which means I'm not sure the UI will work at all on internet explorer
j

Jeff Williams

01/27/2021, 8:57 PM
Ok. I am using IE just as a stop gap. I have to use a self-signed cert for now while I am in a POC sandbox but will be getting a full cert as we move out of POC. What you have told me helps me to move forward though so I will focus on working out the kinks with the cert....
n

nicholas

01/27/2021, 8:58 PM
Got it - I wonder if spinning up the UI container as normal and adding an nginx proxy in front of your setup might be easier than trying to use a custom container
But I'm sure you've got more of a handle on that 👍
j

Jeff Williams

01/27/2021, 9:00 PM
Funny you mention that as that was a line of discussion today. From a "maintainability" perspective, using an nginx proxy might be the better way to go.
n

nicholas

01/27/2021, 9:01 PM
Yeah almost guaranteed - one problem with custom images is that upgrading your version of the stack is hard
j

Jeff Williams

01/27/2021, 10:32 PM
@nicholas - I have resolved the self-cert problem and am able to open the UI in the latest version of Chrome now. It is still blank though. Looking at this screen shot, obviously these are some internal blockers going on. So you have any suggestions or do you think going the proxy route we discussed would solve these issues?
n

nicholas

01/27/2021, 10:34 PM
Ah it looks like you've pulled the master branch of the UI, instead of the latest release
Which means you won't have the required config for a pending auth change that we haven't released yet.
j

Jeff Williams

01/27/2021, 10:35 PM
So pulling from the different branch should solve this issue?
n

nicholas

01/27/2021, 10:35 PM
It'll resolve at least 1 of those big errors; you'll want to use the latest release branch instead of master
j

Jeff Williams

01/27/2021, 10:37 PM
There is not a UI branch named "latest-release"
n

nicholas

01/27/2021, 10:38 PM
The latest release meaning the most recent release of the UI, the releases can be found here: https://github.com/PrefectHQ/ui/releases (along with the commit refs and tags)
j

Jeff Williams

01/27/2021, 10:39 PM
Sorry. Was looking in the wrong place on github.... 😆
Hmmm, standing up that proxy is starting to sound like a much better idea.... 🤣
n

nicholas

01/27/2021, 11:37 PM
Prefect Server is a big stack! Definitely not a trivial thing to deploy in a production environment with modifications. Have you looked into using Cloud at all? You'll find you don't have to worry about any of the overhead of managing your orchestration stack, you'll only need to worry about compute.
j

Jeff Williams

01/28/2021, 1:44 PM
Personally, I have wondered about using Cloud. However, that decision is made above my paygrade. I don't know if the concern is from a compliance perspective or a cost perspective. But I think, IMO, it should be something to consider.
n

nicholas

01/28/2021, 2:37 PM
Understood - If you or someone at your company is curious, we’ve got a team they can talk to to discuss compliance and cost! Cloud’s hybrid model typically satisfies all compliance concerns from my experience.
j

Jeff Williams

01/28/2021, 3:40 PM
Thanks. Regarding the compliance concerns, that is what I understood from the reading / research I have been doing while working with Prefect...
@nicholas - So I am trying to get the proxy version configured up. Since I have made so many attempts, I removed all the Prefect docker images that were previously downloaded and I have made a pretty the config.toml file very basic with just the items you mentioned in your GCP article. When I run the prefect server start, it will pull new images (which I want) and then try to stand things up. When the UI tries to come up, it throws an error as shown below. Ideas?
n

nicholas

01/28/2021, 6:45 PM
yeah it looks like you've still got a container running on that port
you could try something like:
docker stop $(docker ps -aq)
to stop all running containers and try that aagain
j

Jeff Williams

01/28/2021, 6:49 PM
Let me look again. I thought I shut them all down..
n

nicholas

01/28/2021, 6:49 PM
it's always possible it's not a container using the port but something is using that port
j

Jeff Williams

01/28/2021, 6:51 PM
Ahhh, I see what it is. I am trying to use an nginx docker container as the proxy and it is also binding to the 8080 port so that it can redirect traffic from 443 to 8080 locally.
@nicholas - Wanted you to know that I finally got everything configured properly and have the Prefect UI up and running. I stood up a docker container running nginx in front of the request to handle the security aspects that were a concern. Thanks for all your help on achieving this. Now we can start playing with it in more depth...
n

nicholas

01/29/2021, 7:37 PM
Amazing! Well done 😄
j

Jeff Williams

01/29/2021, 7:39 PM
The key thing for our set up was to connect the nginx container to the prefect-server network as it is starting up. This removed the need for the nginx container to bind to port 8080. Now we can present a secured external connection and forward that to the UI via the private prefect-server network.
n

nicholas

01/29/2021, 8:02 PM
That makes sense! 🙂
j

Jeff Williams

02/01/2021, 5:40 PM
@nicholas - I am having trouble verifying that Apollo server is working properly with our configuration. I can ping the Apollo server instance so I know that it is alive. I tried running a curl command but didn't get anything useful back, but that is probably because I don't really know what to query for. Do you have anything tips on how to validate the Apollo server it functioning properly? Is there something from a CLI perspective that I can run to get some feedback? I am looking through the docs but nothing really stands out to me. Any help would be appreciated... 🙂
n

nicholas

02/01/2021, 6:00 PM
Hi @Jeff Williams - you can perform a health check on the Apollo server the http://localhost:4201/health endpoint - you should be able to access that using the same routing mechanism you're already using
j

Jeff Williams

02/01/2021, 6:15 PM
Thanks @nicholas. When I did that I get a "Connection refused" so I think I need to revisit the config.toml file. I looked at the startup logs and it "SAYS" that the Apollo server started and is living on 4200 so I would thing the 4201 should be working as well.
n

nicholas

02/01/2021, 6:16 PM
Hm, give that a try on 4200/health
j

Jeff Williams

02/01/2021, 6:18 PM
That gave me a reply of "GET query missing"....
n

nicholas

02/01/2021, 6:19 PM
i'm a little surprised that's the response but that tells you the server is up and running because apollo is expecting some data
j

Jeff Williams

02/01/2021, 6:21 PM
Well that is good then. In the UI, when I go to Team | Projects, I get a blank page and an occasional blue progress bar at the top just below the menu. Nothing else. I was thinking that was possibly because the Apollo server wasn't working properly. Any idease?
n

nicholas

02/01/2021, 6:23 PM
Is there anything in the javascript console?
j

Jeff Williams

02/01/2021, 6:26 PM
Yes. That looks like the key to the mystery. More fun / configuration on my part to deal with my environment it appears.... 🤣🤣🤣 I will work on this and see if I can get it resolved. Thanks for point me in the right direction....
n

nicholas

02/01/2021, 6:26 PM
What's the error you're getting? It's possible the issue isn't in your environment configuration if it's just that page that's not loading correctly
j

Jeff Williams

02/01/2021, 6:30 PM
It is a corporate policy that is suppressing the non-secure request.
n

nicholas

02/01/2021, 6:31 PM
Ahhh Ok that makes sense
j

Jeff Williams

02/02/2021, 8:55 PM
@nicholas - Good afternoon. Here is today's question installment... 🙂 In working through the console error / information messages in my browser, I am seeing several calls to "external" entities that would probably be fine in most environments. However, it got me to wondering if there was a "fully contained" version of Prefect that didn't make external calls. Do you happen to know?
n

nicholas

02/02/2021, 9:00 PM
Not just yet but we hope to provide some airgapped solution in the near-er future!
z

Zanie

02/02/2021, 9:01 PM
There is a tracking issue for that somewhere -- it might be internal.
😆 1
^
n

nicholas

02/02/2021, 9:01 PM
jinx
j

Jeff Williams

02/02/2021, 9:02 PM
LOL - That is good to hear. When you say near-er future, are you thinking Q1 of this year, Q2, etc? You may not know right now, but it potentially impacts some of our internal decisions.
n

nicholas

02/02/2021, 9:03 PM
i would expect that closer to the end of Q1, if not sooner
j

Jeff Williams

02/02/2021, 9:03 PM
Oh, that is great! That would play into our plans nicely then....
@nicholas - I am trying to run a simple CLI command as I am still having issues with the GUI, again, probably internal constraints. I ran
prefect create project "project name" -d "project description"
and it puked on me with a message about "No connection adapters were found for <server_ip:4200/graphql>. I am unsure where I was supposed to define / configure the connection adapters. Any direction on what what I did wrong here?
n

nicholas

02/05/2021, 6:10 PM
Hm yeah my guess is your server_ip is missing the protocol
you'll need http:// or https://
j

Jeff Williams

02/05/2021, 6:30 PM
That helped. I am getting a different error now, again, environment configuration related. 🤣
n

nicholas

02/05/2021, 6:37 PM
yay 🙂