Has anyone come across a case where you build a flow locally, and the register step is configured to register to a remote server (private prefect instance on aws, not prefect cloud). The output give me a url where the flow should be registered on that server, but there's nothing in the UI..

Hi @Darragh! Can you share a little more information with me? 1. The message you get back after calling

flow.register

2. Are you building a docker container locally? If so, where are you pushing the image? 3. Make sure that apollo’s port is exposed on your Prefect Server instance 4. Would you mind sharing your

.prefect/config.toml

configuration that points to your remote Prefect Server instance on your machine?

Hey, Yeah I across that in a previous post, I'm going to tes tit out now!

Great! Let us know how it goes 😄

i have somewhat similar issue, setting server server.ui.graphql_url (or ENV var) did help in case of local server, but not working from another VM - I got an error: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: /graphql/alpha in prefect/client/client.py", line 640. After setting cloud.graphql=server_url the registration succeeds, however nothing appears in UI

Hi @Simon Basin, when did that error appear? On server startup?

no errors on server startup... just no flows shown on UI

Hi @Simon Basin, take a look at: https://medium.com/the-prefect-blog/prefect-server-101-deploying-to-google-cloud-platform-47354b16afe2

I am running server locally, no cloud

Hi guys, reading that article it seems like Nicholas says you need to set the server.ui graphql_url to be the machine ip, rather than localhost, is that right?

If you’re deploying in the cloud, that’s correct

So the .prefect/config.toml, that’s separate to the one installed by

pip install prefect

right?

that depends on what you’re trying to configure

Alright, probably best for me to go through my setup then, make sure I’m asking the right question… • Prefect stack deployed on AWS EC2 - do I need to do anything to the config.toml installed with the package? • Does the EC2 Prefect need a

~./prefect/config.toml

[presumably this overrides the one instaled with Prefect?] with it’s own IP in it for the graphql endpoint? • Local machine - flow.register() - what do I need to alter in my local Prefect installation to get that flow registered with http://EC2_BOX ??

Hi @Darragh great questions! 1. You don't need to do anything with the

config.toml

installed with the package, you'll only need to modify the one at `~/.prefect/config.toml`; this may not be created automatically so you may have to

mkdir ~/.prefect && touch ~/.prefect/config.toml

2. You're correct that your EC2 Prefect installation will need its own

~/.prefect/config.toml

(the one you just created), in which you'll override the

graphql_url

in the way @Dylan posted above. As you said, this will be the IP address of the EC2 you've installed the stack on (basically it'll direct the UI to itself) 3. On your local machine, you'll run

prefect backend server

and point prefect to your remote installation. You can do this by running

export PREFECT___CLOUD__API=<http://EC2_BOX>_

Ahhhh I see, so those should be the only steps I need, and ideally I shouldn’t have to make any changes to the config.toml installed with prefect, but rather update the specific parameters remotely and locally.

Exactly, 2 things are involved: pointing your local prefect to your remote installation, and letting the UI you installed remotely know where it's installed so that when it runs client-side (your browser), it knows where to look for its API!

thanks @nickolas! I will try to use

PREFECT__CLOUD__API=_

Oops, quick edit on that, it's

PREFECT___CLOUD___API

, Slack likes to interpret my underscores as italics <.<

yes i figured...:)

Hm, unsure what you mean about that @Simon Basin - what's the error you're running into?

I can see flow on UI, but agent fails to pick it up...

Ah, ok great! We're making progress.

yes it looks like container from which I have registered the flow, while agent is running on the same host as server for now (eventually I would like to run it in docker container too)

Ah ok, so before you start that agent (since it's on the machine), you can run

prefect backend server

and it'll know to poll

localhost:4200/graphql

so only server is using ~/.prefect/flows ?

Gotcha - would you mind starting your agent in the foreground with the

--verbose

flag? That way we can see the output

i am running w/ verbose and capturing all output

@Simon Basin would you mind sharing all of the output above the first

Waiting flow flow runs...

line?

basically it says no flows found

Hey @Simon Basin, can you share the command you’re using to start the agent and the labels on the flow you’re attempting to run, please?

Sorry, the logs above that, they should look something like this:

[2020-05-14 16:40:55,812] DEBUG - Version Group Tests | Verbose logs enabled
[2020-05-14 16:40:55,812] DEBUG - Version Group Tests | Environment variables: []
[2020-05-14 16:40:55,812] DEBUG - Version Group Tests | Max polls: None
[2020-05-14 16:40:55,812] DEBUG - Version Group Tests | Agent address: 
[2020-05-14 16:40:55,812] DEBUG - Version Group Tests | Prefect backend: server
[2020-05-14 16:40:56,155] DEBUG - Version Group Tests | Import paths: []
[2020-05-14 16:40:56,155] DEBUG - Version Group Tests | Show flow logs: False

 ____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2020-05-14 16:40:56,155] INFO - Version Group Tests | Starting LocalAgent with labels ['Nicholas-Little-Mac.local', 'NicholaittleMac.fios-router.home', 'azure-flow-storage', 'gcs-flow-storage', 's3-flow-storage']
[2020-05-14 16:40:56,156] INFO - Version Group Tests | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-05-14 16:40:56,156] INFO - Version Group Tests | Agent connecting to the Prefect API at <https://localhost:4200/graphql>

ok when I run agent with container's label I got an error on the client: with open(str(fpath), "rb") as f: FileNotFoundError: [Errno 2] No such file or directory: '/root/.prefect/flows/test-flow.prefect' [2020-05-14 165259,979] INFO - agent | Process PID 37505 returned non-zero exit code

can you share the command you’re using to start the agent and the labels on the flow you’re attempting to run, please?

thanks @Dylan & @nickolas! Adding labels did help to make some progress. I am now working on running agent inside a docker container. I need it to have our python runtime environment and our code-base - can I use DockerAgent for that or I need to setup agent myself?

Do you need the agent to have your python runtime environment and codebase, or does the flow need access to those?

as I have extended Task and Flow classes both agent and flow seems need to access to specific runtime/code

That’s totally fine! The distinction here is that we need that code to be present in your Flow’s container, not your agent’s machine

I’m still getting nowhere with the registration, so I have 2 follow up questions: • is there any logging I can turn on in both the remote server instance, and the local regsitration command • If I’m updating the graphql endpoint in the remote server config.toml, is it the eth0 IP, the public IP, some other..? And definitely not loclhost?

@Darragh it's however you access the remote server from outside. Normally this will be the public IP, appended with

:4200/graphql

; that port (4200) will need to be opened to outside traffic. As for logging,

perfect server start

has a lot of output, including network traffic to the APi, which should prove useful. Unfortunately I don't believe the

.register

command has any additional logging

I have the pulibc ip in the toml, so I'll see I'll trap the logs and see what it gives me. Thanks!

@Darragh make sure you've included the http protocol with the IP, and you'll need to restart the server for changes there to apply!

Yeah I've gotten all those, I'm beginning to think it just hates me 😍

😬 😭

Networking is hard

Don’t tempt me 😂

First thought is that last line

378 Flow: <http://localhost:8080/flow/87adb8df-c3d1-4745-af0b-31752f1091b3>

has localhost in the URL

Tbh this could be a red herring, let me check if flow registration output respects host overrides

Yeah that’s why I was using the overrides you guys supplied way back up the thread, doesn’t seem to be reflected in the flow. Not 100% sure if the request made it to the server either

Hm so this gives us a bit of info at least. 1) Since there was output, the request made it to the server wherever you've set

PREFECT_CLOUD_API

. 2) Since this is using server, the output of

flow.regsiter

is based on the

prefect.config.server.ui.endpoint

var, which defaults to

localhost

. This isn't a true reflection of where the UI is deployed though, since Prefect has no way of knowing unless that variable is explicitly set.

So am I best then to override the value in the file, or use an evnironment variable to override it?

I'd recommend to override using the config file, just so it sticks

Ok, then I have 2 outstanding issues to fix, override the config and test the remote register 👍

Nice! Well done!

I need to do some tidying up, but if you guys want my solution as a sample for how to do GitLab/Containerized builds of flows and deploying Prefect & Flows to AWS I’d be happy to share [there were some uniquely interesting twists to this damn thing] ?? Might save some other poor shmuck some pain 🙂

Definitely! That would be a great addition, I think we've got a place in the works specifically for this, lemme see if I can find