https://prefect.io logo
a

Akshey

09/15/2023, 2:06 PM
Hi Community, I need help with the Prefect server, I can run the flow in the terminal, but flow runs are not visible in the Prefect UI. I can even see the logs in the terminal. Also, I am running the Prefect server on EC2. Steps Followed: β€’ prefect server start β€’
prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>
β€’ run the python script
Copy code
09:40:56.066 | INFO    | prefect.engine - Created flow run 'jasper-dachshund' for flow 'main_flow'
09:40:56.070 | INFO    | Flow run 'jasper-dachshund' - View at <http://127.0.0.1:4200/flow-runs/flow-run/27bd6913-5bf8-440c-958c-5ebfceea0b7b>
09:40:56.176 | INFO    | Flow run 'jasper-dachshund' - Created task run 'load_combine_dataset-0' for task 'load_combine_dataset'
09:40:56.178 | INFO    | Flow run 'jasper-dachshund' - Executing 'load_combine_dataset-0' immediately...
09:41:27.748 | INFO    | Task run 'load_combine_dataset-0' - Finished in state Completed()
09:41:30.110 | INFO    | Flow run 'jasper-dachshund' - Created task run 'lexical_features-0' for task 'lexical_features'
09:41:30.112 | INFO    | Flow run 'jasper-dachshund' - Executing 'lexical_features-0' immediately...
09:42:34.146 | INFO    | Task run 'lexical_features-0' - Finished in state Completed()
09:42:38.077 | INFO    | Flow run 'jasper-dachshund' - Created task run 'data_preprocess-0' for task 'data_preprocess'
09:42:38.078 | INFO    | Flow run 'jasper-dachshund' - Executing 'data_preprocess-0' immediately...
09:46:31.694 | INFO    | Task run 'data_preprocess-0' - Finished in state Completed()
09:46:31.759 | INFO    | Flow run 'jasper-dachshund' - Finished in state Completed('All states completed.')
Can anyone please help to debug this?
i

Islam Otmani

09/15/2023, 9:51 PM
Hi @Akshey, If you run
prefect profile ls
, and
prefect profile inspect
, are the values matching your expectation?
i.e.
PREFECT_API_URL=<http://127.0.0.1:4200/api>
?
a

Akshey

09/15/2023, 9:52 PM
Thanks, @Islam Otmani for your response. Let me verify
i

Islam Otmani

09/15/2023, 9:52 PM
πŸ‘
a

Akshey

09/15/2023, 9:56 PM
Screenshot 2023-09-15 at 5.56.46 PM.png
Yes, it looks like both values are same
i

Islam Otmani

09/15/2023, 9:57 PM
Great, that rules out my suspicion.
a

Akshey

09/15/2023, 9:58 PM
The first time I tested it was working fine but now I have tried everything and flow runs are not appearing on the UI.
i

Islam Otmani

09/15/2023, 10:02 PM
Do you see anything when you then run:
Copy code
curl '<http://127.0.0.1:4200/api/work_pools/filter>'  -H 'Content-Type: application/json'  --data-raw '{}'
Or perhaps better yet, any flow runs, via:
Copy code
curl '<http://127.0.0.1:4200/api/flow_runs/count>'  -H 'Content-Type: application/json'  --data-raw '{}'
a

Akshey

09/15/2023, 10:04 PM
Copy code
[{"id":"a1feda9c-ad3f-4c9d-8d18-ea7ab61427d0","created":"2023-08-11T18:47:58.348000+00:00","updated":"2023-08-11T18:47:58.348000+00:00","name":"default-agent-pool","description":null,"type":"prefect-agent","base_job_template":{},"is_paused":false,"concurrency_limit":null,"default_queue_id":"eadb5e37-d20e-4d36-af62-511fc7f3b930"},{"id":"837881d4-0795-4e82-8401-281ccd731700","created":"2023-08-18T23:04:52.405877+00:00","updated":"2023-08-18T23:04:52.417000+00:00","name":"zoompool","description":null,"type":"process","base_job_template":{"job_configuration":{"command":"{{ command }}","env":"{{ env }}","labels":"{{ labels }}","name":"{{ name }}","stream_output":"{{ stream_output }}","working_dir":"{{ working_dir }}"},"variables":{"type":"object","properties":{"name":{"title":"Name","description":"Name given to infrastructure created by a worker.","type":"string"},"env":{"title":"Environment Variables","description":"Environment variables to set when starting a flow run.","type":"object","additionalProperties":{"type":"string"}},"labels":{"title":"Labels","description":"Labels applied to infrastructure created by a worker.","type":"object","additionalProperties":{"type":"string"}},"command":{"title":"Command","description":"The command to use when starting a flow run. In most cases, this should be left blank and the command will be automatically generated by the worker.","type":"string"},"stream_output":{"title":"Stream Output","description":"If enabled, workers will stream output from flow run processes to local standard output.","default":true,"type":"boolean"},"working_dir":{"title":"Working Directory","description":"If provided, workers will open flow run processes within the specified path as the working directory. Otherwise, a temporary directory will be created.","type":"string","format":"path"}}}},"is_paused":false,"concurrency_limit":null,"default_queue_id":"c0c5a6c5-b6cb-4706-9fe6-df9783fbd741"}](prefect-ops)
Count: 38
i

Islam Otmani

09/15/2023, 10:05 PM
Great, so the runs and data is there
How are you trying to access the UI if I may ask?
From your computer to EC2 I mean, via the machine's public IP?
a

Akshey

09/15/2023, 10:07 PM
I have port forwarding on as I am using Visual Studio, I am running the Python script and visiting http://127.0.0.1:4201/flow-runs on my browser.
πŸ‘ 1
I cannot see any runs although I can see the logs and completion message on my terminal.
i

Islam Otmani

09/15/2023, 10:08 PM
I'm suspecting you might have multiple instances running, The above was port
4200
and this is
4201
ah nvm
just saw the full image, ignore me πŸ™‚
a

Akshey

09/15/2023, 10:09 PM
No worries!
i

Islam Otmani

09/15/2023, 10:09 PM
Just to confirm, did you run the CURL count command above from within an EC2 shell right?
a

Akshey

09/15/2023, 10:10 PM
Yes
I have EC2 instance connected to my local through ssh
πŸ‘ 1
i

Islam Otmani

09/15/2023, 10:11 PM
From your computer, try it again via the forwarded port please:
Copy code
curl '<http://127.0.0.1:4201/api/flow_runs/count>'  -H 'Content-Type: application/json'  --data-raw '{}'
a

Akshey

09/15/2023, 10:12 PM
curl: (7) Failed to connect to 127.0.0.1 port 4201 after 0 ms: Connection refused
4201 is the forwarded port. The port open on EC2 is 4200
i

Islam Otmani

09/15/2023, 10:13 PM
Yes I wanted to test things from the forwarded port side this time around
Since all is well up until then
πŸ‘ 1
a

Akshey

09/15/2023, 10:13 PM
I reran the python script and the count incremented by 1. It's 39 now
i

Islam Otmani

09/15/2023, 10:16 PM
The above error tells us something is wrong with the forwarded connection, try close/reopen VSCode so it re-establishes the forwarding?
Alternatively, open Chrome Inspector (Dev Tools), and look under the Network tab at the traffic for any connection errors
a

Akshey

09/15/2023, 10:18 PM
Sure let me try restarting the VScode
πŸ‘ 1
Screenshot 2023-09-15 at 6.20.00 PM.png
Also, I have successfully re-established connection after restarting VScode
i

Islam Otmani

09/15/2023, 10:22 PM
Great. Those Console messages look fine/unrelated. Here's my Network tab for comparison:
a

Akshey

09/15/2023, 10:24 PM
Screenshot 2023-09-15 at 6.24.31 PM.png
i

Islam Otmani

09/15/2023, 10:25 PM
If you click on
count
(the first row), and 'Preview', do you see your expected count of
39
by any chance?
a

Akshey

09/15/2023, 10:27 PM
Screenshot 2023-09-15 at 6.27.01 PM.png
I can only see 0
i

Islam Otmani

09/15/2023, 10:28 PM
Aha.. This tells us port forwarding is fine, everything is working fine on this side of things also. Please double check your the EC2 instance you're SSH'ed into, and the instance one you're forwarding the port is the same, as all it smells of multiple instances running πŸ™‚
Since the same API call is returning different values when ran from within SSH and via port forward
a

Akshey

09/15/2023, 10:29 PM
I have only one EC2 instance running
πŸ€” 1
I tested the same on other VM machine too
Screenshot 2023-09-15 at 6.30.03 PM.png
Is it working fine on your end?
I noticed there were three new releases in the last two weeks. Is there any potential bug?
i

Islam Otmani

09/15/2023, 10:37 PM
Yes, all is well from our end as far as I can tell, something like this would break a lot of tests also I'd imagine. So to recap this case: β€’ Server running on EC2, HTTP calls to UI from within SSH works and we see expected data. β€’ Port forwarding the same instance via VS, we get a different response.
a

Akshey

09/15/2023, 10:39 PM
I will try updating VScode as well if that's the problem
i

Islam Otmani

09/15/2023, 10:43 PM
I don't think that's necessary, as the UI wouldn't be accessible in the first place if there was a networking issue. My next suggestion is to try port forwarding via the SSH linux command, rather than VSCode, and connect to the UI from this other port. I.e. via a SSH tunnel.
a

Akshey

09/15/2023, 10:44 PM
Copy code
ssh -L [LOCAL_IP:]LOCAL_PORT:DESTINATION:DESTINATION_PORT [USER@]SSH_SERVER
πŸ‘ 1
Ok I will give it a shot using this command
Trying now
I will keep you posted! Thanks a lot for all your help @Islam Otmani
i

Islam Otmani

09/15/2023, 11:14 PM
It should look something like this as you mentioned, then point your browser to http://localhost:4300
ssh -L 4300:myec2:4200 myuser@myec2
You got it my friend, we're very close πŸ™‚πŸ‘
a

Akshey

09/15/2023, 11:15 PM
I wish it works as I have to present a POC of Prefect integration to my ML code
i

Islam Otmani

09/15/2023, 11:17 PM
Oh I see, happy to jump on a call to speed things up if it still doesn't work
πŸ™ 1
a

Akshey

09/15/2023, 11:17 PM
Permission Denied
when I am trying to ssh
Do you know if I have to set the inbound rule for port 4300 as well?
i

Islam Otmani

09/15/2023, 11:19 PM
nope, it just uses port 22 for SSH as usual, and forwards traffic through that
Do you get permission denied only when you try port forwarding? or also with a regular ssh ?
(maybe VSCode is passing an SSH key)
a

Akshey

09/15/2023, 11:23 PM
Permission denied (publickey).
i

Islam Otmani

09/15/2023, 11:24 PM
Aha, it's having trouble logging in
a

Akshey

09/15/2023, 11:24 PM
Yes
i

Islam Otmani

09/15/2023, 11:27 PM
You should find the required detail for the host, key, portforwarding etc in VSCode settings > Remote SSH extension, since that's working fine.
a

Akshey

09/15/2023, 11:28 PM
Ok let me check
Screenshot 2023-09-15 at 7.38.45 PM.png
I tested the VS code port forwarding for MLflow as well and it's working fine
i

Islam Otmani

09/16/2023, 12:10 AM
Hmm it's a little hard to debug from out here it turns out. But obv your ssh config is passed correctly somehow. And I've no doubt now that it's forwarding correctly, since the UI works. I just cannot prove that it's connected to another VM or prefect server instance, otherwise we'd not see the same API call returning different values.
a

Akshey

09/16/2023, 4:29 PM
I doubt it could be an issue on the Prefect side during port forwarding
As I mentioned, the entire process worked smoothly when I tested it 3-4 weeks back
@Islam Otmani To prove it's not connected to another VM, I will be happy to jump on a quick call to validate things if it's possible
i

Islam Otmani

09/17/2023, 4:30 AM
Sure, sounds good. Tomorrow anytime or Monday works pacific hours for me if that works. It’ll be easier to troubleshoot rather than me throwing the next guess at you :)
a

Akshey

09/17/2023, 4:36 AM
I agree thanks a lot! We can connect sometime tomorrow.