Hi all. I hope I am in the right channel. I am trying to set up Prefect in a GCP account using a micro-VM as discussed in this article: https://medium.com/the-prefect-blog/prefect-server-101-deploying-to-google-cloud-platform-47354b16afe2. What is happening is that I an getting time out errors on the spin up of tmp_apollo_1 and tmp_towel_1 that say "(UnixHTTPConnectionPool (host='localhost', port=None): Read timed out. (read timeout=60)". Then there is some reference to a possible slow network. Everything else up to that point has worked fine, per the article. When I ping my localhost from the ssh window, it responds immediately, so I don't think it is a slow network issue. I could possibly be a configuration issue as the environment I am working in has to be tightly controlled due to compliance issues. But I think I have the appropriate Firewall Rules in place to allow the right things. Again, using the article as my road map. Any help would be greatly appreciated.

Perhaps take a look at https://github.com/docker/compose/issues/3927

Thanks. I have read through the thread and will see if my issue gets resolved.

Interesting. I’ll let the author of the article know.

I am going to try going back to the same size VM as mentioned in the article, but use some of the other changes that I made when I moved to the larger VM. This is still a POC for me and I will be happy to share the results later today.

Great! Thanks

Hi @Jeff Williams - sorry for the confusion on that, I realize there's a screenshot of the instance creation view in GCP, but that wasn't intended to be an instruction on how large/small a machine to use. Prefect Server currently consists of 7 nodes, each running as containers through Docker, and so is a nontrivial application stack. Because of this, you'll need to use a larger VM type than

micro

and probably even larger still if you plan to scale out your workloads (and don't forget to attach a volume!)

Good to know @nicholas. The screenshot did lead me to believe that was a pseudo-recommendation. My bad...

I'd updated the description on that picture to avoid confusion; thanks for bringing that to our attention 🙂

I will make a point to review the Architecture link you provided as well... 😉

Hi @Jeff Williams - when you say you're having trouble getting the UI to show up, do you mean that you can't access the UI at all or that it isn't connecting to the Apollo server correctly?

When I try to use my browser window to get to the UI, as directed, I does not connect. Working within the confines of our firewall rules, I have opened it up to allow any IP address from within our organization using any protocol to any port to connect to the GCP VM. That is when I started looking at the on-screen output and noticing references to the server being at 0.0.0.0 instead of the IP address assigned in the config.toml file.

Hi @Jeff Williams - unfortunately I don't. If you can't get to the UI, that tells me your local machine can't see the address over which you're serving it. Given you've got some internal firewall rules that could be preventing you from accessing your deployment, I'm not sure how much I can diagnose on this end.

@nicholas - Understandable. Once I get it figured out, if it is relevant, I will share with you the solution in case it is helpful for someone else.

Much appreciated, if you have any questions about any of the Server internals that might be helpful lmk!

@nicholas - Yes, and it was user error evidently so I deleted the evidence.... 🤣🤣🤣🤣 I am going to try and move forward now....

Haha no shame in that! Happens to all of us 🙂

@nicholas - I have been working on a few different solutions and what I have settled on is to run the server with the --no-gui option so that all other components will load up & start. Then, I have downloaded the UI git repo and built a "custom" image where I am adding a self-signed cert (for now) to allow the UI to run on port 443 (SSL - A requirement for my industry) I have also made the change in the nginx.conf to listen on 443 rather than 8080. It SEEMS to be working, in that the webpage will load and change the title on the tab to "Prefect". However, it is not showing any data. I simply get a blank screen. When I issue the "docker container logs <container>" command, I am getting log data but nothing tells me one way or the other if everything is talking or not talking to each other. How can I tell? Do you have any troubleshooting tips for me?

Hi @Jeff Williams - can you provide a screenshot of what you're seeing, including the javascript console?

@nicholas - Do you mean an F12 console "DOM Explorer" screenshot or something from the docker logs?

If you press F12 (or cmd+option+i on mac) and then navigate to the

Console

tab, you'll see a javascript developer console 🙂

Actually, I see a message under the <noscript> tag saying that JavaScript is not enabled. Let me see if that is my issue and I will get back with you...

Hm, that's a standard tag on most SPAs, if it's not displaying in the browser then you should be fine

Well, if that is the case, my F12 doesn't show a JavaScript developer console. I see options for: "DOM Explorer", Console, Debugger, Network, Performance, Memory, and Emulation.

Right,

Console

is what you're looking for

All it lists is 3 - "i" icons with "Current window" information.

Could you screenshot what you're seeing?

Yes. It is on a different system that I can't load Slack onto so I have to do some hoop jumping.... 🙂

Hm ok, there are a few issues I'm seeing: 1) you've got a certificate error, which means most browsers won't serve anything and 2) we only support modern browsers which means I'm not sure the UI will work at all on internet explorer

Ok. I am using IE just as a stop gap. I have to use a self-signed cert for now while I am in a POC sandbox but will be getting a full cert as we move out of POC. What you have told me helps me to move forward though so I will focus on working out the kinks with the cert....

Got it - I wonder if spinning up the UI container as normal and adding an nginx proxy in front of your setup might be easier than trying to use a custom container

Funny you mention that as that was a line of discussion today. From a "maintainability" perspective, using an nginx proxy might be the better way to go.

Yeah almost guaranteed - one problem with custom images is that upgrading your version of the stack is hard

@nicholas - I have resolved the self-cert problem and am able to open the UI in the latest version of Chrome now. It is still blank though. Looking at this screen shot, obviously these are some internal blockers going on. So you have any suggestions or do you think going the proxy route we discussed would solve these issues?

Ah it looks like you've pulled the master branch of the UI, instead of the latest release

So pulling from the different branch should solve this issue?

It'll resolve at least 1 of those big errors; you'll want to use the latest release branch instead of master

There is not a UI branch named "latest-release"

The latest release meaning the most recent release of the UI, the releases can be found here: https://github.com/PrefectHQ/ui/releases (along with the commit refs and tags)

Sorry. Was looking in the wrong place on github.... 😆

Prefect Server is a big stack! Definitely not a trivial thing to deploy in a production environment with modifications. Have you looked into using Cloud at all? You'll find you don't have to worry about any of the overhead of managing your orchestration stack, you'll only need to worry about compute.

Personally, I have wondered about using Cloud. However, that decision is made above my paygrade. I don't know if the concern is from a compliance perspective or a cost perspective. But I think, IMO, it should be something to consider.

Understood - If you or someone at your company is curious, we’ve got a team they can talk to to discuss compliance and cost! Cloud’s hybrid model typically satisfies all compliance concerns from my experience.

Thanks. Regarding the compliance concerns, that is what I understood from the reading / research I have been doing while working with Prefect...

yeah it looks like you've still got a container running on that port

Let me look again. I thought I shut them all down..

it's always possible it's not a container using the port but something is using that port

Ahhh, I see what it is. I am trying to use an nginx docker container as the proxy and it is also binding to the 8080 port so that it can redirect traffic from 443 to 8080 locally.

Amazing! Well done 😄

The key thing for our set up was to connect the nginx container to the prefect-server network as it is starting up. This removed the need for the nginx container to bind to port 8080. Now we can present a secured external connection and forward that to the UI via the private prefect-server network.

That makes sense! 🙂

@nicholas - I am having trouble verifying that Apollo server is working properly with our configuration. I can ping the Apollo server instance so I know that it is alive. I tried running a curl command but didn't get anything useful back, but that is probably because I don't really know what to query for. Do you have anything tips on how to validate the Apollo server it functioning properly? Is there something from a CLI perspective that I can run to get some feedback? I am looking through the docs but nothing really stands out to me. Any help would be appreciated... 🙂

Hi @Jeff Williams - you can perform a health check on the Apollo server the http://localhost:4201/health endpoint - you should be able to access that using the same routing mechanism you're already using

Thanks @nicholas. When I did that I get a "Connection refused" so I think I need to revisit the config.toml file. I looked at the startup logs and it "SAYS" that the Apollo server started and is living on 4200 so I would thing the 4201 should be working as well.

Hm, give that a try on 4200/health

That gave me a reply of "GET query missing"....

i'm a little surprised that's the response but that tells you the server is up and running because apollo is expecting some data

Well that is good then. In the UI, when I go to Team | Projects, I get a blank page and an occasional blue progress bar at the top just below the menu. Nothing else. I was thinking that was possibly because the Apollo server wasn't working properly. Any idease?

Is there anything in the javascript console?

Yes. That looks like the key to the mystery. More fun / configuration on my part to deal with my environment it appears.... 🤣🤣🤣 I will work on this and see if I can get it resolved. Thanks for point me in the right direction....

What's the error you're getting? It's possible the issue isn't in your environment configuration if it's just that page that's not loading correctly

It is a corporate policy that is suppressing the non-secure request.

Ahhh Ok that makes sense

@nicholas - Good afternoon. Here is today's question installment... 🙂 In working through the console error / information messages in my browser, I am seeing several calls to "external" entities that would probably be fine in most environments. However, it got me to wondering if there was a "fully contained" version of Prefect that didn't make external calls. Do you happen to know?

Not just yet but we hope to provide some airgapped solution in the near-er future!

There is a tracking issue for that somewhere -- it might be internal.

jinx

LOL - That is good to hear. When you say near-er future, are you thinking Q1 of this year, Q2, etc? You may not know right now, but it potentially impacts some of our internal decisions.

i would expect that closer to the end of Q1, if not sooner

Oh, that is great! That would play into our plans nicely then....

Hm yeah my guess is your server_ip is missing the protocol

That helped. I am getting a different error now, again, environment configuration related. 🤣

yay 🙂