https://prefect.io logo
Join Slack
Powered by
Hi Community, I need help with the Prefect server,...
# ask-community
a
Hi Community, I need help with the Prefect server, I can run the flow in the terminal, but flow runs are not visible in the Prefect UI. I can even see the logs in the terminal. Also, I am running the Prefect server on EC2. Steps Followed: • prefect server start • 
prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>
• run the python script
Copy code
09:40:56.066 | INFO    | prefect.engine - Created flow run 'jasper-dachshund' for flow 'main_flow'
09:40:56.070 | INFO    | Flow run 'jasper-dachshund' - View at <http://127.0.0.1:4200/flow-runs/flow-run/27bd6913-5bf8-440c-958c-5ebfceea0b7b>
09:40:56.176 | INFO    | Flow run 'jasper-dachshund' - Created task run 'load_combine_dataset-0' for task 'load_combine_dataset'
09:40:56.178 | INFO    | Flow run 'jasper-dachshund' - Executing 'load_combine_dataset-0' immediately...
09:41:27.748 | INFO    | Task run 'load_combine_dataset-0' - Finished in state Completed()
09:41:30.110 | INFO    | Flow run 'jasper-dachshund' - Created task run 'lexical_features-0' for task 'lexical_features'
09:41:30.112 | INFO    | Flow run 'jasper-dachshund' - Executing 'lexical_features-0' immediately...
09:42:34.146 | INFO    | Task run 'lexical_features-0' - Finished in state Completed()
09:42:38.077 | INFO    | Flow run 'jasper-dachshund' - Created task run 'data_preprocess-0' for task 'data_preprocess'
09:42:38.078 | INFO    | Flow run 'jasper-dachshund' - Executing 'data_preprocess-0' immediately...
09:46:31.694 | INFO    | Task run 'data_preprocess-0' - Finished in state Completed()
09:46:31.759 | INFO    | Flow run 'jasper-dachshund' - Finished in state Completed('All states completed.')
Can anyone please help to debug this?
i
Hi @Akshey, If you run 
prefect profile ls
, and 
prefect profile inspect
, are the values matching your expectation?
i.e. 
PREFECT_API_URL=<http://127.0.0.1:4200/api>
?
a
Thanks, @Islam Otmani for your response. Let me verify
i
👍
a
Yes, it looks like both values are same
i
Great, that rules out my suspicion.
a
The first time I tested it was working fine but now I have tried everything and flow runs are not appearing on the UI.
i
Do you see anything when you then run:
Copy code
curl '<http://127.0.0.1:4200/api/work_pools/filter>'  -H 'Content-Type: application/json'  --data-raw '{}'
Or perhaps better yet, any flow runs, via:
Copy code
curl '<http://127.0.0.1:4200/api/flow_runs/count>'  -H 'Content-Type: application/json'  --data-raw '{}'
a
Copy code
[{"id":"a1feda9c-ad3f-4c9d-8d18-ea7ab61427d0","created":"2023-08-11T18:47:58.348000+00:00","updated":"2023-08-11T18:47:58.348000+00:00","name":"default-agent-pool","description":null,"type":"prefect-agent","base_job_template":{},"is_paused":false,"concurrency_limit":null,"default_queue_id":"eadb5e37-d20e-4d36-af62-511fc7f3b930"},{"id":"837881d4-0795-4e82-8401-281ccd731700","created":"2023-08-18T23:04:52.405877+00:00","updated":"2023-08-18T23:04:52.417000+00:00","name":"zoompool","description":null,"type":"process","base_job_template":{"job_configuration":{"command":"{{ command }}","env":"{{ env }}","labels":"{{ labels }}","name":"{{ name }}","stream_output":"{{ stream_output }}","working_dir":"{{ working_dir }}"},"variables":{"type":"object","properties":{"name":{"title":"Name","description":"Name given to infrastructure created by a worker.","type":"string"},"env":{"title":"Environment Variables","description":"Environment variables to set when starting a flow run.","type":"object","additionalProperties":{"type":"string"}},"labels":{"title":"Labels","description":"Labels applied to infrastructure created by a worker.","type":"object","additionalProperties":{"type":"string"}},"command":{"title":"Command","description":"The command to use when starting a flow run. In most cases, this should be left blank and the command will be automatically generated by the worker.","type":"string"},"stream_output":{"title":"Stream Output","description":"If enabled, workers will stream output from flow run processes to local standard output.","default":true,"type":"boolean"},"working_dir":{"title":"Working Directory","description":"If provided, workers will open flow run processes within the specified path as the working directory. Otherwise, a temporary directory will be created.","type":"string","format":"path"}}}},"is_paused":false,"concurrency_limit":null,"default_queue_id":"c0c5a6c5-b6cb-4706-9fe6-df9783fbd741"}](prefect-ops)
Count: 38
i
Great, so the runs and data is there
How are you trying to access the UI if I may ask?
From your computer to EC2 I mean, via the machine's public IP?
a
I have port forwarding on as I am using Visual Studio, I am running the Python script and visiting http://127.0.0.1:4201/flow-runs on my browser.
👍 1
I cannot see any runs although I can see the logs and completion message on my terminal.
i
I'm suspecting you might have multiple instances running, The above was port 
4200
and this is 
4201
ah nvm
just saw the full image, ignore me 🙂
a
No worries!
i
Just to confirm, did you run the CURL count command above from within an EC2 shell right?
a
Yes
I have EC2 instance connected to my local through ssh
👍 1
i
From your computer, try it again via the forwarded port please:
Copy code
curl '<http://127.0.0.1:4201/api/flow_runs/count>'  -H 'Content-Type: application/json'  --data-raw '{}'
a
curl: (7) Failed to connect to 127.0.0.1 port 4201 after 0 ms: Connection refused
4201 is the forwarded port. The port open on EC2 is 4200
i
Yes I wanted to test things from the forwarded port side this time around
Since all is well up until then
👍 1
a
I reran the python script and the count incremented by 1. It's 39 now
i
The above error tells us something is wrong with the forwarded connection, try close/reopen VSCode so it re-establishes the forwarding?
Alternatively, open Chrome Inspector (Dev Tools), and look under the Network tab at the traffic for any connection errors
a
Sure let me try restarting the VScode
👍 1
Also, I have successfully re-established connection after restarting VScode
i
Great. Those Console messages look fine/unrelated. Here's my Network tab for comparison:
a
i
If you click on 
count
(the first row), and 'Preview', do you see your expected count of 
39
by any chance?
a
I can only see 0
i
Aha.. This tells us port forwarding is fine, everything is working fine on this side of things also. Please double check your the EC2 instance you're SSH'ed into, and the instance one you're forwarding the port is the same, as all it smells of multiple instances running 🙂
Since the same API call is returning different values when ran from within SSH and via port forward
a
I have only one EC2 instance running
🤔 1
I tested the same on other VM machine too
Is it working fine on your end?
I noticed there were three new releases in the last two weeks. Is there any potential bug?
i
Yes, all is well from our end as far as I can tell, something like this would break a lot of tests also I'd imagine. So to recap this case: • Server running on EC2, HTTP calls to UI from within SSH works and we see expected data. • Port forwarding the same instance via VS, we get a different response.
a
I will try updating VScode as well if that's the problem
i
I don't think that's necessary, as the UI wouldn't be accessible in the first place if there was a networking issue. My next suggestion is to try port forwarding via the SSH linux command, rather than VSCode, and connect to the UI from this other port. I.e. via a SSH tunnel.
a
Copy code
ssh -L [LOCAL_IP:]LOCAL_PORT:DESTINATION:DESTINATION_PORT [USER@]SSH_SERVER
👍 1
Ok I will give it a shot using this command
Trying now
I will keep you posted! Thanks a lot for all your help @Islam Otmani
i
It should look something like this as you mentioned, then point your browser to http://localhost:4300 
ssh -L 4300:myec2:4200 myuser@myec2
You got it my friend, we're very close 🙂👍
a
I wish it works as I have to present a POC of Prefect integration to my ML code
i
Oh I see, happy to jump on a call to speed things up if it still doesn't work
🙏 1
a
Permission Denied
when I am trying to ssh
Do you know if I have to set the inbound rule for port 4300 as well?
i
nope, it just uses port 22 for SSH as usual, and forwards traffic through that
Do you get permission denied only when you try port forwarding? or also with a regular ssh ?
(maybe VSCode is passing an SSH key)
a
Permission denied (publickey).
i
Aha, it's having trouble logging in
a
Yes
i
You should find the required detail for the host, key, portforwarding etc in VSCode settings > Remote SSH extension, since that's working fine.
a
Ok let me check
I tested the VS code port forwarding for MLflow as well and it's working fine
i
Hmm it's a little hard to debug from out here it turns out. But obv your ssh config is passed correctly somehow. And I've no doubt now that it's forwarding correctly, since the UI works. I just cannot prove that it's connected to another VM or prefect server instance, otherwise we'd not see the same API call returning different values.
a
I doubt it could be an issue on the Prefect side during port forwarding
As I mentioned, the entire process worked smoothly when I tested it 3-4 weeks back
@Islam Otmani To prove it's not connected to another VM, I will be happy to jump on a quick call to validate things if it's possible
i
Sure, sounds good. Tomorrow anytime or Monday works pacific hours for me if that works. It’ll be easier to troubleshoot rather than me throwing the next guess at you :)
a
I agree thanks a lot! We can connect sometime tomorrow.
3 Views
Open in Slack
PreviousNext
Powered by