Channels
#prefect-community
Title
a
Akshey
09/15/2023, 2:06 PMHi Community, I need help with the Prefect server, I can run the flow in the terminal, but flow runs are not visible in the Prefect UI. I can even see the logs in the terminal. Also, I am running the Prefect server on EC2.
Steps Followed:
• prefect server start
•
prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api> Copy code
09:40:56.066 | INFO | prefect.engine - Created flow run 'jasper-dachshund' for flow 'main_flow'
09:40:56.070 | INFO | Flow run 'jasper-dachshund' - View at <http://127.0.0.1:4200/flow-runs/flow-run/27bd6913-5bf8-440c-958c-5ebfceea0b7b>
09:40:56.176 | INFO | Flow run 'jasper-dachshund' - Created task run 'load_combine_dataset-0' for task 'load_combine_dataset'
09:40:56.178 | INFO | Flow run 'jasper-dachshund' - Executing 'load_combine_dataset-0' immediately...
09:41:27.748 | INFO | Task run 'load_combine_dataset-0' - Finished in state Completed()
09:41:30.110 | INFO | Flow run 'jasper-dachshund' - Created task run 'lexical_features-0' for task 'lexical_features'
09:41:30.112 | INFO | Flow run 'jasper-dachshund' - Executing 'lexical_features-0' immediately...
09:42:34.146 | INFO | Task run 'lexical_features-0' - Finished in state Completed()
09:42:38.077 | INFO | Flow run 'jasper-dachshund' - Created task run 'data_preprocess-0' for task 'data_preprocess'
09:42:38.078 | INFO | Flow run 'jasper-dachshund' - Executing 'data_preprocess-0' immediately...
09:46:31.694 | INFO | Task run 'data_preprocess-0' - Finished in state Completed()
09:46:31.759 | INFO | Flow run 'jasper-dachshund' - Finished in state Completed('All states completed.')i
Islam Otmani
09/15/2023, 9:51 PMHi @Akshey,
If you run
prefect profile lsprefect profile inspecti.e.
PREFECT_API_URL=<http://127.0.0.1:4200/api>a
Akshey
09/15/2023, 9:52 PMThanks, @Islam Otmani for your response. Let me verify
i
Islam Otmani
09/15/2023, 9:52 PM👍
a
Akshey
09/15/2023, 9:56 PMScreenshot 2023-09-15 at 5.56.46 PM.png
Yes, it looks like both values are same
i
Islam Otmani
09/15/2023, 9:57 PMGreat, that rules out my suspicion.
a
Akshey
09/15/2023, 9:58 PMThe first time I tested it was working fine but now I have tried everything and flow runs are not appearing on the UI.
i
Islam Otmani
09/15/2023, 10:02 PMDo you see anything when you then run:
Copy code
curl '<http://127.0.0.1:4200/api/work_pools/filter>' -H 'Content-Type: application/json' --data-raw '{}'Or perhaps better yet, any flow runs, via:
Copy code
curl '<http://127.0.0.1:4200/api/flow_runs/count>' -H 'Content-Type: application/json' --data-raw '{}'a
Akshey
09/15/2023, 10:04 PM Copy code
[{"id":"a1feda9c-ad3f-4c9d-8d18-ea7ab61427d0","created":"2023-08-11T18:47:58.348000+00:00","updated":"2023-08-11T18:47:58.348000+00:00","name":"default-agent-pool","description":null,"type":"prefect-agent","base_job_template":{},"is_paused":false,"concurrency_limit":null,"default_queue_id":"eadb5e37-d20e-4d36-af62-511fc7f3b930"},{"id":"837881d4-0795-4e82-8401-281ccd731700","created":"2023-08-18T23:04:52.405877+00:00","updated":"2023-08-18T23:04:52.417000+00:00","name":"zoompool","description":null,"type":"process","base_job_template":{"job_configuration":{"command":"{{ command }}","env":"{{ env }}","labels":"{{ labels }}","name":"{{ name }}","stream_output":"{{ stream_output }}","working_dir":"{{ working_dir }}"},"variables":{"type":"object","properties":{"name":{"title":"Name","description":"Name given to infrastructure created by a worker.","type":"string"},"env":{"title":"Environment Variables","description":"Environment variables to set when starting a flow run.","type":"object","additionalProperties":{"type":"string"}},"labels":{"title":"Labels","description":"Labels applied to infrastructure created by a worker.","type":"object","additionalProperties":{"type":"string"}},"command":{"title":"Command","description":"The command to use when starting a flow run. In most cases, this should be left blank and the command will be automatically generated by the worker.","type":"string"},"stream_output":{"title":"Stream Output","description":"If enabled, workers will stream output from flow run processes to local standard output.","default":true,"type":"boolean"},"working_dir":{"title":"Working Directory","description":"If provided, workers will open flow run processes within the specified path as the working directory. Otherwise, a temporary directory will be created.","type":"string","format":"path"}}}},"is_paused":false,"concurrency_limit":null,"default_queue_id":"c0c5a6c5-b6cb-4706-9fe6-df9783fbd741"}](prefect-ops)Count: 38
i
Islam Otmani
09/15/2023, 10:05 PMGreat, so the runs and data is there
How are you trying to access the UI if I may ask?
From your computer to EC2 I mean, via the machine's public IP?
a
Akshey
09/15/2023, 10:07 PMI have port forwarding on as I am using Visual Studio, I am running the Python script and visiting http://127.0.0.1:4201/flow-runs on my browser.
👍 1
I cannot see any runs although I can see the logs and completion message on my terminal.
i
Islam Otmani
09/15/2023, 10:08 PMI'm suspecting you might have multiple instances running,
The above was port
42004201ah nvm
just saw the full image, ignore me 🙂
a
Akshey
09/15/2023, 10:09 PMNo worries!
i
Islam Otmani
09/15/2023, 10:09 PMJust to confirm, did you run the CURL count command above from within an EC2 shell right?
a
Akshey
09/15/2023, 10:10 PMYes
I have EC2 instance connected to my local through ssh
👍 1
i
Islam Otmani
09/15/2023, 10:11 PMFrom your computer, try it again via the forwarded port please:
Copy code
curl '<http://127.0.0.1:4201/api/flow_runs/count>' -H 'Content-Type: application/json' --data-raw '{}'a
Akshey
09/15/2023, 10:12 PMcurl: (7) Failed to connect to 127.0.0.1 port 4201 after 0 ms: Connection refused4201 is the forwarded port. The port open on EC2 is 4200
i
Islam Otmani
09/15/2023, 10:13 PMYes I wanted to test things from the forwarded port side this time around
Since all is well up until then
👍 1
a
Akshey
09/15/2023, 10:13 PMI reran the python script and the count incremented by 1. It's 39 now
i
Islam Otmani
09/15/2023, 10:16 PMThe above error tells us something is wrong with the forwarded connection, try close/reopen VSCode so it re-establishes the forwarding?
Alternatively, open Chrome Inspector (Dev Tools), and look under the Network tab at the traffic for any connection errors
a
Akshey
09/15/2023, 10:18 PMSure let me try restarting the VScode
👍 1
Screenshot 2023-09-15 at 6.20.00 PM.png
Also, I have successfully re-established connection after restarting VScode
i
Islam Otmani
09/15/2023, 10:22 PMGreat.
Those Console messages look fine/unrelated.
Here's my Network tab for comparison:
a
Akshey
09/15/2023, 10:24 PMScreenshot 2023-09-15 at 6.24.31 PM.png
i
Islam Otmani
09/15/2023, 10:25 PMIf you click on
count39a
Akshey
09/15/2023, 10:27 PMScreenshot 2023-09-15 at 6.27.01 PM.png
I can only see 0
i
Islam Otmani
09/15/2023, 10:28 PMAha.. This tells us port forwarding is fine, everything is working fine on this side of things also.
Please double check your the EC2 instance you're SSH'ed into, and the instance one you're forwarding the port is the same, as all it smells of multiple instances running 🙂
Since the same API call is returning different values when ran from within SSH and via port forward
a
Akshey
09/15/2023, 10:29 PMI have only one EC2 instance running
🤔 1
I tested the same on other VM machine too
Screenshot 2023-09-15 at 6.30.03 PM.png
Is it working fine on your end?
I noticed there were three new releases in the last two weeks. Is there any potential bug?
i
Islam Otmani
09/15/2023, 10:37 PMYes, all is well from our end as far as I can tell, something like this would break a lot of tests also I'd imagine.
So to recap this case:
• Server running on EC2, HTTP calls to UI from within SSH works and we see expected data.
• Port forwarding the same instance via VS, we get a different response.
a
Akshey
09/15/2023, 10:39 PMI will try updating VScode as well if that's the problem
i
Islam Otmani
09/15/2023, 10:43 PMI don't think that's necessary, as the UI wouldn't be accessible in the first place if there was a networking issue.
My next suggestion is to try port forwarding via the SSH linux command, rather than VSCode, and connect to the UI from this other port. I.e. via a SSH tunnel.
a
Akshey
09/15/2023, 10:44 PM Copy code
ssh -L [LOCAL_IP:]LOCAL_PORT:DESTINATION:DESTINATION_PORT [USER@]SSH_SERVER👍 1
Ok I will give it a shot using this command
Trying now
I will keep you posted! Thanks a lot for all your help @Islam Otmani
i
Islam Otmani
09/15/2023, 11:14 PMIt should look something like this as you mentioned, then point your browser to http://localhost:4300
ssh -L 4300:myec2:4200 myuser@myec2You got it my friend, we're very close 🙂👍
a
Akshey
09/15/2023, 11:15 PMI wish it works as I have to present a POC of Prefect integration to my ML code
i
Islam Otmani
09/15/2023, 11:17 PMOh I see, happy to jump on a call to speed things up if it still doesn't work
🙏 1
a
Akshey
09/15/2023, 11:17 PMPermission DeniedDo you know if I have to set the inbound rule for port 4300 as well?
i
Islam Otmani
09/15/2023, 11:19 PMnope, it just uses port 22 for SSH as usual, and forwards traffic through that
Do you get permission denied only when you try port forwarding?
or also with a regular ssh ?
(maybe VSCode is passing an SSH key)
a
Akshey
09/15/2023, 11:23 PMPermission denied (publickey).i
Islam Otmani
09/15/2023, 11:24 PMAha, it's having trouble logging in
a
Akshey
09/15/2023, 11:24 PMYes
i
Islam Otmani
09/15/2023, 11:27 PMYou should find the required detail for the host, key, portforwarding etc in VSCode settings > Remote SSH extension, since that's working fine.
a
Akshey
09/15/2023, 11:28 PMOk let me check
Screenshot 2023-09-15 at 7.38.45 PM.png
I tested the VS code port forwarding for MLflow as well and it's working fine
i
Islam Otmani
09/16/2023, 12:10 AMHmm it's a little hard to debug from out here it turns out. But obv your ssh config is passed correctly somehow.
And I've no doubt now that it's forwarding correctly, since the UI works. I just cannot prove that it's connected to another VM or prefect server instance, otherwise we'd not see the same API call returning different values.
a
Akshey
09/16/2023, 4:29 PMI doubt it could be an issue on the Prefect side during port forwarding
As I mentioned, the entire process worked smoothly when I tested it 3-4 weeks back
@Islam Otmani To prove it's not connected to another VM, I will be happy to jump on a quick call to validate things if it's possible
i
Islam Otmani
09/17/2023, 4:30 AMSure, sounds good. Tomorrow anytime or Monday works pacific hours for me if that works. It’ll be easier to troubleshoot rather than me throwing the next guess at you :)
a
Akshey
09/17/2023, 4:36 AMI agree thanks a lot! We can connect sometime tomorrow.