Has anyone come across a case where you build a fl...
# prefect-community
Has anyone come across a case where you build a flow locally, and the register step is configured to register to a remote server (private prefect instance on aws, not prefect cloud). The output give me a url where the flow should be registered on that server, but there's nothing in the UI..
👀 1
Hi @Darragh! Can you share a little more information with me? 1. The message you get back after calling
2. Are you building a docker container locally? If so, where are you pushing the image? 3. Make sure that apollo’s port is exposed on your Prefect Server instance 4. Would you mind sharing your
configuration that points to your remote Prefect Server instance on your machine?
We also see an issue where the API is exposed properly, but the UI isn’t communicating with it. On the machine running your Prefect Server, make sure the UI is configured to point at the correct endpoint:
Copy code
    graphql_url = "INSERT_ENDPOINT_HERE:4200/graphql"
Hey, Yeah I across that in a previous post, I'm going to tes tit out now!
Great! Let us know how it goes 😄
i have somewhat similar issue, setting server server.ui.graphql_url (or ENV var) did help in case of local server, but not working from another VM - I got an error: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: /graphql/alpha in prefect/client/client.py", line 640. After setting cloud.graphql=server_url the registration succeeds, however nothing appears in UI
Hi @Simon Basin, when did that error appear? On server startup?
no errors on server startup... just no flows shown on UI
Not sure if you’re on GCP or not, but Nicholas goes over how to set everything up in good detail
If you’re unsure if the UI is connecting to the correct endpoint, you can open your browser’s devtools and look for failing network requests
I am running server locally, no cloud
I decided to try changing cloud.graphql setting after looking into Client c-tor.
Hi guys, reading that article it seems like Nicholas says you need to set the server.ui graphql_url to be the machine ip, rather than localhost, is that right?
If you’re deploying in the cloud, that’s correct
So the .prefect/config.toml, that’s separate to the one installed by
pip install prefect
Another clarification - where am I creating that file? On my AWS installation, or on the machine I’m trying to do registration from?
that depends on what you’re trying to configure
For server configurations such as
that interact with prefect server, you’ll need to update the
on the machine where you’re running Prefect Server
and in order for those changes to take effect, you’ll need to restart Prefect Server
For configuration related to registering flows, you set that on the machine that’s trying to build and register that flow
Alright, probably best for me to go through my setup then, make sure I’m asking the right question… • Prefect stack deployed on AWS EC2 - do I need to do anything to the config.toml installed with the package? • Does the EC2 Prefect need a
[presumably this overrides the one instaled with Prefect?] with it’s own IP in it for the graphql endpoint? • Local machine - flow.register() - what do I need to alter in my local Prefect installation to get that flow registered with http://EC2_BOX ??
These are probably pretty basic, but I’m still noobish 🙂
Hi @Darragh great questions! 1. You don't need to do anything with the
installed with the package, you'll only need to modify the one at `~/.prefect/config.toml`; this may not be created automatically so you may have to
mkdir ~/.prefect && touch ~/.prefect/config.toml
2. You're correct that your EC2 Prefect installation will need its own
(the one you just created), in which you'll override the
in the way @Dylan posted above. As you said, this will be the IP address of the EC2 you've installed the stack on (basically it'll direct the UI to itself) 3. On your local machine, you'll run
prefect backend server
and point prefect to your remote installation. You can do this by running
export PREFECT___CLOUD__API=<http://EC2_BOX>_
Ahhhh I see, so those should be the only steps I need, and ideally I shouldn’t have to make any changes to the config.toml installed with prefect, but rather update the specific parameters remotely and locally.
Thanks for that Nicholas, I’ll try it out shortly and let you know!
🚀 1
Exactly, 2 things are involved: pointing your local prefect to your remote installation, and letting the UI you installed remotely know where it's installed so that when it runs client-side (your browser), it knows where to look for its API!
@Simon Basin I think those steps should help you as well, some things to note: • you need to include the protocol schema in your endpoint url, i.e.:
instead of
• if you're spinning up the Prefect Server docker network anywhere but locally, including in a VM or a remote machine with AWS, GCP, Azure etc, you'll need to let the Prefect Server installation know where it's installed so that when you access the UI in your browser (the client), it knows where to look for its API
thanks @nickolas! I will try to use
Oops, quick edit on that, it's
, Slack likes to interpret my underscores as italics <.<
yes i figured...:)
😬 1
looks like for registration it doesn't help, however I got a bit further with PREFECT__CLOUD___GRAPHQL=http://server:4200/graphql
now I think I need to adjust local storage location : which property controls that?
Hm, unsure what you mean about that @Simon Basin - what's the error you're running into?
I can see flow on UI, but agent fails to pick it up...
when I click on RUN
Ah, ok great! We're making progress.
So if you look on the Flow page, you'll see a "labels" section in the details tile for the flow
Sec I'll grab a secreenshot
Your local agent will need to be started with all of the labels you find there (and you'll want to start it with the same environment variables you registered the flow with, so it knows where your API is)
so in my case above, i'd start my agent like this:
Copy code
prefect agent start --name "Super Awesome Local Agent" --label Nicholas-Little-Mac.local
yes it looks like container from which I have registered the flow, while agent is running on the same host as server for now (eventually I would like to run it in docker container too)
Ah ok, so before you start that agent (since it's on the machine), you can run
prefect backend server
and it'll know to poll
so only server is using ~/.prefect/flows ?
1. I have started both server and agent from the same bash session putting them in background with &
Gotcha - would you mind starting your agent in the foreground with the
flag? That way we can see the output
i am running w/ verbose and capturing all output
i have run flow registration from inside a docker container
@Simon Basin would you mind sharing all of the output above the first
Waiting flow flow runs...
basically it says no flows found
[2020-05-14 164657,357] DEBUG - agent | Next query for flow runs in 10.0 seconds [2020-05-14 164707,357] DEBUG - agent | Querying for flow runs [2020-05-14 164707,376] DEBUG - agent | No flow runs found
Hey @Simon Basin, can you share the command you’re using to start the agent and the labels on the flow you’re attempting to run, please?
Sorry, the logs above that, they should look something like this:
Copy code
[2020-05-14 16:40:55,812] DEBUG - Version Group Tests | Verbose logs enabled
[2020-05-14 16:40:55,812] DEBUG - Version Group Tests | Environment variables: []
[2020-05-14 16:40:55,812] DEBUG - Version Group Tests | Max polls: None
[2020-05-14 16:40:55,812] DEBUG - Version Group Tests | Agent address: 
[2020-05-14 16:40:55,812] DEBUG - Version Group Tests | Prefect backend: server
[2020-05-14 16:40:56,155] DEBUG - Version Group Tests | Import paths: []
[2020-05-14 16:40:56,155] DEBUG - Version Group Tests | Show flow logs: False

 ____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|

[2020-05-14 16:40:56,155] INFO - Version Group Tests | Starting LocalAgent with labels ['Nicholas-Little-Mac.local', 'NicholaittleMac.fios-router.home', 'azure-flow-storage', 'gcs-flow-storage', 's3-flow-storage']
[2020-05-14 16:40:56,156] INFO - Version Group Tests | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-05-14 16:40:56,156] INFO - Version Group Tests | Agent connecting to the Prefect API at <https://localhost:4200/graphql>
ok when I run agent with container's label I got an error on the client: with open(str(fpath), "rb") as f: FileNotFoundError: [Errno 2] No such file or directory: '/root/.prefect/flows/test-flow.prefect' [2020-05-14 165259,979] INFO - agent | Process PID 37505 returned non-zero exit code
can you share the command you’re using to start the agent and the labels on the flow you’re attempting to run, please?
thanks @Dylan & @nickolas! Adding labels did help to make some progress. I am now working on running agent inside a docker container. I need it to have our python runtime environment and our code-base - can I use DockerAgent for that or I need to setup agent myself?
Do you need the agent to have your python runtime environment and codebase, or does the flow need access to those?
If you use Docker storage, the DockerAgent will pull and run the container for you
as I have extended Task and Flow classes both agent and flow seems need to access to specific runtime/code
That’s totally fine! The distinction here is that we need that code to be present in your Flow’s container, not your agent’s machine
I’m still getting nowhere with the registration, so I have 2 follow up questions: • is there any logging I can turn on in both the remote server instance, and the local regsitration command • If I’m updating the graphql endpoint in the remote server config.toml, is it the eth0 IP, the public IP, some other..? And definitely not loclhost?
@Darragh it's however you access the remote server from outside. Normally this will be the public IP, appended with
; that port (4200) will need to be opened to outside traffic. As for logging,
perfect server start
has a lot of output, including network traffic to the APi, which should prove useful. Unfortunately I don't believe the
command has any additional logging
I have the pulibc ip in the toml, so I'll see I'll trap the logs and see what it gives me. Thanks!
@Darragh make sure you've included the http protocol with the IP, and you'll need to restart the server for changes there to apply!
and before registration, you'll want to set the environment variables on your local machine that I mentioned above to point to your remote server
Yeah I've gotten all those, I'm beginning to think it just hates me 😍
😬 😭
Networking is hard
If you want to put it down and come back later, you can always give https://cloud.prefect.io a shot! There’s a free tier
Don’t tempt me 😂
😂 1
Hey guys, back again. I have an update on my connection to AWS Prefect problem, hopefully it makes sense to you. I’ve added the variable override for the local machine so that when it tries to register it’s calling out to AWS, but it’s getting the following back: echo $PREFECT__CLOUD__API 373 374 $ echo $PREFECT__CLOUD__GRAPHQL 375 376 $ python3 packages/flows/definitions/test.py 377 Result check: OK 378 Flow: http://localhost:8080/flow/87adb8df-c3d1-4745-af0b-31752f1091b3 Any thoughts on what setting I’m missing here?
First thought is that last line
378 Flow: <http://localhost:8080/flow/87adb8df-c3d1-4745-af0b-31752f1091b3>
has localhost in the URL
That feels wrong to me? Since the UI should be represented by
http://{IP dotted quad}/flow/{uuid}
I think
Thoughts @nicholas @Kyle Moon-Wright?
Tbh this could be a red herring, let me check if flow registration output respects host overrides
Yeah that’s why I was using the overrides you guys supplied way back up the thread, doesn’t seem to be reflected in the flow. Not 100% sure if the request made it to the server either
By the by, the above output is from my containerized build machine, those same overrides on my local machine [laptop] produce the an output URL that points to the expected endpoint
Hm so this gives us a bit of info at least. 1) Since there was output, the request made it to the server wherever you've set
. 2) Since this is using server, the output of
is based on the
var, which defaults to
. This isn't a true reflection of where the UI is deployed though, since Prefect has no way of knowing unless that variable is explicitly set.
@Darragh - what's the issue you're seeing currently? Just from what you posted above, I'm not seeing any errors or anything
So am I best then to override the value in the file, or use an evnironment variable to override it?
Basically I gave the register the remote location to post to, and it gave me a localhost URL
I'd recommend to override using the config file, just so it sticks
As for that, it's not necessarily an issue, just that the Prefect Client doesn't know where your UI is. If you were to visit (assuming that's where you've deployed Prefect Server and you set the
variable before starting it), you should be able to see the flow there
Ok, then I have 2 outstanding issues to fix, override the config and test the remote register 👍
😄 1
*Qapla*’!!!! Using the various variables I finally got it all lined up and registering a basic flow to AWS.
🎉 2
Thank you muchly for all your help!!
Nice! Well done!
hattip 1
I need to do some tidying up, but if you guys want my solution as a sample for how to do GitLab/Containerized builds of flows and deploying Prefect & Flows to AWS I’d be happy to share [there were some uniquely interesting twists to this damn thing] ?? Might save some other poor shmuck some pain 🙂
Definitely! That would be a great addition, I think we've got a place in the works specifically for this, lemme see if I can find