Hi Community, I need help with the Prefect server, I can run the flow in the terminal, but flow runs...

Akshey

09/15/2023, 2:06 PM

Hi Community, I need help with the Prefect server, I can run the flow in the terminal, but flow runs are not visible in the Prefect UI. I can even see the logs in the terminal. Also, I am running the Prefect server on EC2. Steps Followed: • prefect server start •

prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>

• run the python script

Copy code

09:40:56.066 | INFO    | prefect.engine - Created flow run 'jasper-dachshund' for flow 'main_flow'
09:40:56.070 | INFO    | Flow run 'jasper-dachshund' - View at <http://127.0.0.1:4200/flow-runs/flow-run/27bd6913-5bf8-440c-958c-5ebfceea0b7b>
09:40:56.176 | INFO    | Flow run 'jasper-dachshund' - Created task run 'load_combine_dataset-0' for task 'load_combine_dataset'
09:40:56.178 | INFO    | Flow run 'jasper-dachshund' - Executing 'load_combine_dataset-0' immediately...
09:41:27.748 | INFO    | Task run 'load_combine_dataset-0' - Finished in state Completed()
09:41:30.110 | INFO    | Flow run 'jasper-dachshund' - Created task run 'lexical_features-0' for task 'lexical_features'
09:41:30.112 | INFO    | Flow run 'jasper-dachshund' - Executing 'lexical_features-0' immediately...
09:42:34.146 | INFO    | Task run 'lexical_features-0' - Finished in state Completed()
09:42:38.077 | INFO    | Flow run 'jasper-dachshund' - Created task run 'data_preprocess-0' for task 'data_preprocess'
09:42:38.078 | INFO    | Flow run 'jasper-dachshund' - Executing 'data_preprocess-0' immediately...
09:46:31.694 | INFO    | Task run 'data_preprocess-0' - Finished in state Completed()
09:46:31.759 | INFO    | Flow run 'jasper-dachshund' - Finished in state Completed('All states completed.')

Can anyone please help to debug this?

Islam Otmani

09/15/2023, 9:51 PM

Hi @Akshey, If you run

prefect profile ls

, and

prefect profile inspect

, are the values matching your expectation?

Islam Otmani

09/15/2023, 9:51 PM

i.e.

PREFECT_API_URL=<http://127.0.0.1:4200/api>

Akshey

09/15/2023, 9:52 PM

Thanks, @Islam Otmani for your response. Let me verify

Islam Otmani

09/15/2023, 9:52 PM

👍

Akshey

09/15/2023, 9:56 PM

Akshey

09/15/2023, 9:57 PM

Yes, it looks like both values are same

Islam Otmani

09/15/2023, 9:57 PM

Great, that rules out my suspicion.

Akshey

09/15/2023, 9:58 PM

The first time I tested it was working fine but now I have tried everything and flow runs are not appearing on the UI.

Islam Otmani

09/15/2023, 10:02 PM

Do you see anything when you then run:

Islam Otmani

09/15/2023, 10:02 PM

Copy code

curl '<http://127.0.0.1:4200/api/work_pools/filter>'  -H 'Content-Type: application/json'  --data-raw '{}'

Islam Otmani

09/15/2023, 10:04 PM

Or perhaps better yet, any flow runs, via:

Copy code

curl '<http://127.0.0.1:4200/api/flow_runs/count>'  -H 'Content-Type: application/json'  --data-raw '{}'

Akshey

09/15/2023, 10:04 PM

Copy code

[{"id":"a1feda9c-ad3f-4c9d-8d18-ea7ab61427d0","created":"2023-08-11T18:47:58.348000+00:00","updated":"2023-08-11T18:47:58.348000+00:00","name":"default-agent-pool","description":null,"type":"prefect-agent","base_job_template":{},"is_paused":false,"concurrency_limit":null,"default_queue_id":"eadb5e37-d20e-4d36-af62-511fc7f3b930"},{"id":"837881d4-0795-4e82-8401-281ccd731700","created":"2023-08-18T23:04:52.405877+00:00","updated":"2023-08-18T23:04:52.417000+00:00","name":"zoompool","description":null,"type":"process","base_job_template":{"job_configuration":{"command":"{{ command }}","env":"{{ env }}","labels":"{{ labels }}","name":"{{ name }}","stream_output":"{{ stream_output }}","working_dir":"{{ working_dir }}"},"variables":{"type":"object","properties":{"name":{"title":"Name","description":"Name given to infrastructure created by a worker.","type":"string"},"env":{"title":"Environment Variables","description":"Environment variables to set when starting a flow run.","type":"object","additionalProperties":{"type":"string"}},"labels":{"title":"Labels","description":"Labels applied to infrastructure created by a worker.","type":"object","additionalProperties":{"type":"string"}},"command":{"title":"Command","description":"The command to use when starting a flow run. In most cases, this should be left blank and the command will be automatically generated by the worker.","type":"string"},"stream_output":{"title":"Stream Output","description":"If enabled, workers will stream output from flow run processes to local standard output.","default":true,"type":"boolean"},"working_dir":{"title":"Working Directory","description":"If provided, workers will open flow run processes within the specified path as the working directory. Otherwise, a temporary directory will be created.","type":"string","format":"path"}}}},"is_paused":false,"concurrency_limit":null,"default_queue_id":"c0c5a6c5-b6cb-4706-9fe6-df9783fbd741"}](prefect-ops)

Akshey

09/15/2023, 10:04 PM

Count: 38

Islam Otmani

09/15/2023, 10:05 PM

Great, so the runs and data is there

Islam Otmani

09/15/2023, 10:05 PM

How are you trying to access the UI if I may ask?

Islam Otmani

09/15/2023, 10:07 PM

From your computer to EC2 I mean, via the machine's public IP?

Akshey

09/15/2023, 10:07 PM

I have port forwarding on as I am using Visual Studio, I am running the Python script and visiting http://127.0.0.1:4201/flow-runs on my browser.

👍 1

Akshey

09/15/2023, 10:08 PM

I cannot see any runs although I can see the logs and completion message on my terminal.

Islam Otmani

09/15/2023, 10:08 PM

I'm suspecting you might have multiple instances running, The above was port

and this is

Islam Otmani

09/15/2023, 10:08 PM

ah nvm

Islam Otmani

09/15/2023, 10:08 PM

just saw the full image, ignore me 🙂

Akshey

09/15/2023, 10:09 PM

No worries!

Islam Otmani

09/15/2023, 10:09 PM

Just to confirm, did you run the CURL count command above from within an EC2 shell right?

Akshey

09/15/2023, 10:10 PM

Yes

Akshey

09/15/2023, 10:10 PM

I have EC2 instance connected to my local through ssh

👍 1

Islam Otmani

09/15/2023, 10:11 PM

From your computer, try it again via the forwarded port please:

Copy code

curl '<http://127.0.0.1:4201/api/flow_runs/count>'  -H 'Content-Type: application/json'  --data-raw '{}'

Akshey

09/15/2023, 10:12 PM

curl: (7) Failed to connect to 127.0.0.1 port 4201 after 0 ms: Connection refused

Akshey

09/15/2023, 10:12 PM

4201 is the forwarded port. The port open on EC2 is 4200

Islam Otmani

09/15/2023, 10:13 PM

Yes I wanted to test things from the forwarded port side this time around

Islam Otmani

09/15/2023, 10:13 PM

Since all is well up until then

👍 1

Akshey

09/15/2023, 10:13 PM

I reran the python script and the count incremented by 1. It's 39 now

Islam Otmani

09/15/2023, 10:16 PM

The above error tells us something is wrong with the forwarded connection, try close/reopen VSCode so it re-establishes the forwarding?

Islam Otmani

09/15/2023, 10:17 PM

Alternatively, open Chrome Inspector (Dev Tools), and look under the Network tab at the traffic for any connection errors

Akshey

09/15/2023, 10:18 PM

Sure let me try restarting the VScode

👍 1

Akshey

09/15/2023, 10:20 PM

Akshey

09/15/2023, 10:21 PM

Also, I have successfully re-established connection after restarting VScode

Islam Otmani

09/15/2023, 10:22 PM

Great. Those Console messages look fine/unrelated. Here's my Network tab for comparison:

Akshey

09/15/2023, 10:24 PM

Islam Otmani

09/15/2023, 10:25 PM

If you click on

count

(the first row), and 'Preview', do you see your expected count of

by any chance?

Akshey

09/15/2023, 10:27 PM

Akshey

09/15/2023, 10:27 PM

I can only see 0

Islam Otmani

09/15/2023, 10:28 PM

Aha.. This tells us port forwarding is fine, everything is working fine on this side of things also. Please double check your the EC2 instance you're SSH'ed into, and the instance one you're forwarding the port is the same, as all it smells of multiple instances running 🙂

Islam Otmani

09/15/2023, 10:28 PM

Since the same API call is returning different values when ran from within SSH and via port forward

Akshey

09/15/2023, 10:29 PM

I have only one EC2 instance running

🤔 1

Akshey

09/15/2023, 10:29 PM

I tested the same on other VM machine too

Akshey

09/15/2023, 10:30 PM

Akshey

09/15/2023, 10:32 PM

Is it working fine on your end?

Akshey

09/15/2023, 10:35 PM

I noticed there were three new releases in the last two weeks. Is there any potential bug?

Islam Otmani

09/15/2023, 10:37 PM

Yes, all is well from our end as far as I can tell, something like this would break a lot of tests also I'd imagine. So to recap this case: • Server running on EC2, HTTP calls to UI from within SSH works and we see expected data. • Port forwarding the same instance via VS, we get a different response.

Akshey

09/15/2023, 10:39 PM

I will try updating VScode as well if that's the problem

Islam Otmani

09/15/2023, 10:43 PM

I don't think that's necessary, as the UI wouldn't be accessible in the first place if there was a networking issue. My next suggestion is to try port forwarding via the SSH linux command, rather than VSCode, and connect to the UI from this other port. I.e. via a SSH tunnel.

Akshey

09/15/2023, 10:44 PM

Copy code

ssh -L [LOCAL_IP:]LOCAL_PORT:DESTINATION:DESTINATION_PORT [USER@]SSH_SERVER

👍 1

Akshey

09/15/2023, 10:45 PM

Ok I will give it a shot using this command

Akshey

09/15/2023, 10:48 PM

Trying now

Akshey

09/15/2023, 11:11 PM

I will keep you posted! Thanks a lot for all your help @Islam Otmani

Islam Otmani

09/15/2023, 11:14 PM

It should look something like this as you mentioned, then point your browser to http://localhost:4300

ssh -L 4300:myec2:4200 myuser@myec2

Islam Otmani

09/15/2023, 11:14 PM

You got it my friend, we're very close 🙂👍

Akshey

09/15/2023, 11:15 PM

I wish it works as I have to present a POC of Prefect integration to my ML code

Islam Otmani

09/15/2023, 11:17 PM

Oh I see, happy to jump on a call to speed things up if it still doesn't work

🙏 1

Akshey

09/15/2023, 11:17 PM

Permission Denied

when I am trying to ssh

Akshey

09/15/2023, 11:19 PM

Do you know if I have to set the inbound rule for port 4300 as well?

Islam Otmani

09/15/2023, 11:19 PM

nope, it just uses port 22 for SSH as usual, and forwards traffic through that

Islam Otmani

09/15/2023, 11:21 PM

Do you get permission denied only when you try port forwarding? or also with a regular ssh ?

Islam Otmani

09/15/2023, 11:21 PM

(maybe VSCode is passing an SSH key)

Akshey

09/15/2023, 11:23 PM

Permission denied (publickey).

Islam Otmani

09/15/2023, 11:24 PM

Aha, it's having trouble logging in

Akshey

09/15/2023, 11:24 PM

Yes

Islam Otmani

09/15/2023, 11:27 PM

You should find the required detail for the host, key, portforwarding etc in VSCode settings > Remote SSH extension, since that's working fine.

Akshey

09/15/2023, 11:28 PM

Ok let me check

Akshey

09/15/2023, 11:38 PM

Akshey

09/15/2023, 11:40 PM

I tested the VS code port forwarding for MLflow as well and it's working fine

Islam Otmani

09/16/2023, 12:10 AM

Hmm it's a little hard to debug from out here it turns out. But obv your ssh config is passed correctly somehow. And I've no doubt now that it's forwarding correctly, since the UI works. I just cannot prove that it's connected to another VM or prefect server instance, otherwise we'd not see the same API call returning different values.

Akshey

09/16/2023, 4:29 PM

I doubt it could be an issue on the Prefect side during port forwarding

Akshey

09/16/2023, 4:29 PM

As I mentioned, the entire process worked smoothly when I tested it 3-4 weeks back

Akshey

09/17/2023, 4:24 AM

@Islam Otmani To prove it's not connected to another VM, I will be happy to jump on a quick call to validate things if it's possible

Islam Otmani

09/17/2023, 4:30 AM

Sure, sounds good. Tomorrow anytime or Monday works pacific hours for me if that works. It’ll be easier to troubleshoot rather than me throwing the next guess at you :)

Akshey

09/17/2023, 4:36 AM

I agree thanks a lot! We can connect sometime tomorrow.

2 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.