https://prefect.io logo
Powered by
Title
h

Hugo Polloli

07/21/2021, 9:42 AM
Hi, I'm trying to setup my prefect backend and one agent to be ever-running on my EC2 instance, docker is already automatically starting and since restart is set to true in docker-server's compose.yml, prefect server also automatically starts ever since I launched it once with --detach. But how do I make the agent automatically start on boot too ? according to the docs I could use supervisor with the local agent, is there any alternative using the docker agent ? Should I make my own compose.yml for the agent ?
z

Zanie

07/21/2021, 1:45 PM
@Hugo Polloli You can definitely add an agent container to the docker-compose file. You can run 
prefect server config
to get a configured compose file then modify it to include an agent service or you can just have a separate compose file.
k

Kevin Kho

07/21/2021, 1:46 PM
Hi @Hugo Polloli, I think you can also look into startup scripts.
h

Hugo Polloli

07/21/2021, 1:48 PM
Hi thanks to you both ! I'll look into both of these solutions 🙂
i

Ile Lee

07/24/2021, 2:36 PM
@Hugo Polloli Hi Hugo. How did you get Prefect server to run on your EC2? I've tried Prefect backend server and prefect server start, but I cannot access the UI on the publicIP:8080
k

Kevin Kho

07/24/2021, 4:33 PM
If you SSH into the EC2, can you see it on localhost:8080?
h

Hugo Polloli

07/25/2021, 6:54 AM
If you try the above and can see on localhost:8080, then did you add a security group to your EC2 instance that opens both 8080 and 4200 ports to the public ?
👍 1
:upvote: 1
i

Ile Lee

07/26/2021, 9:23 AM
hi @Hugo Polloli I have tried with opening ports 8080 and 4200 to the public. Funny thing is, on ubuntu it didnt work, but on linux it works. Maybe there's something I'm missing... Is there a post / thread / article you have followed to document the whole process from scratch ?
h

Hugo Polloli

07/26/2021, 9:29 AM
Just in case, here're my inbound rules for the prefect security group I created,
k

Kevin Kho

07/26/2021, 1:44 PM
Hey @Ile Lee, I think what probably happened is the server containers did not start properly. Could you check the logs and see if everything is working? Or check the UI on localhost on the ubuntu server?
i

Ile Lee

07/31/2021, 9:27 AM
@Hugo Polloli @Kevin Kho I've tried again from scratch. There aren't many people who have deployed prefect server on AWS, and since you have, could you correct me if any of my steps are wrong? Sorry I'm still quite new to this field. This is what I've done in the N.Virginia region (us-east-1): Step 1: Created VPC • with CIDR block 10.100.0.0/16 Step 2: Created Subnet • in us-east-1b • with CIDR block 10.100.1.0/24 • enabled auto assign public IPv4 address Step 3: Created Internet Gateway • then attached it to the VPC Step 4: Created Route Table • Added destination 0.0.0.0 to target the internet gateway • In subnet association, added the subnet to explicit subnet association Step 5: Created EC2 • Also created Security Group - added inbound rules TCP 8080 (for prefect UI), TCP 4200 (for graphql), TCP 22 (for SSH) to anyone (0.0.0.0/0) • Downloaded a new key pair Step 6: SSH into the EC2 instance • updated packages • installed docker, docker-compose, prefect (ran sudo service docker start, so docker ps works) Step 7: Created a directory in /home/ec2-user/.prefect Step 8 : Created a config.toml file in /home/ec2-user/.prefect with the following inside it 
[server]
       [server.ui]
       apollo_url = "<http://public-ip:4200/graphql>"
       graphql_url = "<http://public-ip:4200/graphql>"
From there: prefect backend server prefect server start The server starts and says you can visit in localhost:8080 but when I go to my-public-ip:8080, the prefect UI doesn't show I'm sorry for such a long explanation, I just can't figure out why I can't get it running :(
I am using Linux AMI 2
h

Hugo Polloli

07/31/2021, 10:04 AM
Did you apply the security group created at step 5 to the EC2 instance ? See screenshot, then add it. If you already have then sorry but I don't know what else to check 😕
i

Ile Lee

07/31/2021, 1:23 PM
I have 😞 Which AMI are you using?
k

Kevin Kho

07/31/2021, 5:01 PM
Can you confirm localhost 4200 and 8080 are up when you ssh just so we know if it’s a networking issue or issue with spinning those up? Can you check both because the UI image had an issue and was fixed 2 days ago so someone reported 4200 working but now 8080
i

Ile Lee

07/31/2021, 6:57 PM
@Kevin Kho Hi. Sorry I wasn't sure exactly how to do it so I googled it and typed 
nmap localhost
Output was as follows: PORT STATE SERVICE 22/tcp open ssh 3000/tcp open ppp 5432/tcp open postgresql 8080/tcp open http-proxy
It doesn't seem to have port 4200
k

Kevin Kho

07/31/2021, 7:04 PM
Maybe you can try hitting the API with 
prefect get flows
?
Do you also have a browser you can use to open localhost:8080?
i

Ile Lee

07/31/2021, 7:06 PM
by localhost:8080 do you mean the Linux localhost?
Because on my physical computer I go to the browser, type ec2-instance-public-ip:8080
k

Kevin Kho

07/31/2021, 7:08 PM
Yep 8080 is the UI so I’m wondering if you can open the UI with a web browser on the EC2 instance so that we know if it’s a networking issue or the container didn’t spin up right. If you SSH into the instance, 
localhost:8080
should show the browser. Btw, I should just mention because people don’t know this but Prefect Cloud has 10000 free task runs per month also, which is more than enough to get started.
i

Ile Lee

07/31/2021, 7:12 PM
So let's say I've SSH'ed into the EC2
ran Prefect backend server, and prefect server start. lets say my public ec2 instance ip is 3.144.2.10 On firefox I go to 3.144.2.10:8080 This is what it says: Unable to connect Firefox can’t establish a connection to the server at 44.195.83.50:8080. The site could be temporarily unavailable or too busy. Try again in a few moments. If you are unable to load any pages, check your computer’s network connection. If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the Web.
On the SSH terminal I do get the "Welcome to PREFECT SERVER" "Visit http://localhost:8080 to get started, or check the docs at https://docs.prefect.io
I can't reach the UI for some reason
k

Kevin Kho

07/31/2021, 7:18 PM
Can you try clearing the Docker images you have and then doing that again? There was a bug in the UI image fixed 2 days ago. If your UI image tagged as 0.15.3?
Any logs in the UI container?
i

Ile Lee

07/31/2021, 7:23 PM
I see 0.15.3 !
docker images shows core-0.15.3
k

Kevin Kho

07/31/2021, 7:26 PM
Delete those images to force the re-download, and then do the 
prefect server start
again since we fixed the image two days ago so you’ll be sure to pull the right image this time.
i

Ile Lee

07/31/2021, 7:30 PM
So I tried deleted the images using 
docker image prune -a
then ran prefect server start but the images still have the 0.15.3
Am I doing something wrong?
Sorry I'm a bit of a noob
k

Kevin Kho

07/31/2021, 7:45 PM
If you did the 
prefect server stop
, I think prune should work to remove those images. The updates are still tagged as 0.15.3, they were updated. The important thing is you should see new images being downloaded when you do 
prefect server start
i

Ile Lee

07/31/2021, 7:49 PM
Awesome, it is pulling and downloading the images
when i type docker images it still has the tag 0.15.3
k

Kevin Kho

07/31/2021, 7:53 PM
That’s good. I would suggest watching the logs and monitoring if everything is starting successfully, if there’s no errors, you can try the UI at 
localhost:8080
again, and hopefully it works. The UI image logs should also tell you if it started successfully. You can find that container with docker, and then get the logs for that specific container.
i

Ile Lee

07/31/2021, 8:01 PM
towel_1 is throwing errors. Prefect server Lazarus, prefect server Scheduler, prefect server zombie killer
field not found in type query_root
k

Kevin Kho

07/31/2021, 8:03 PM
Could you post your error message here? I think that the UI will at least be up 😅
i

Ile Lee

07/31/2021, 8:07 PM
towel_1     | {"severity": "ERROR", "name": "prefect-server.Lazarus", "message": "Unexpected error: ValueError([{'extensions': {'path': '$.selectionSet.flow_run', 'code': 'validation-failed'}, 'message': 'field \"flow_run\" not found in type: \\'query_root\\''}])", "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n    await self.run_once()\n  File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 37, in run_once\n    return await self.reschedule_flow_runs()\n  File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 89, in reschedule_flow_runs\n    limit=5000,\n  File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n    as_box=not apply_schema,\n  File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n    as_box=as_box,\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 84, in execute\n    raise ValueError(result[\"errors\"])\nValueError: [{'extensions': {'path': '$.selectionSet.flow_run', 'code': 'validation-failed'}, 'message': 'field \"flow_run\" not found in type: \\'query_root\\''}]"}
towel_1     | {"severity": "ERROR", "name": "prefect-server.Scheduler", "message": "Unexpected error: ValueError([{'extensions': {'path': '$.selectionSet.flow', 'code': 'validation-failed'}, 'message': 'field \"flow\" not found in type: \\'query_root\\''}])", "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n    await self.run_once()\n  File \"/prefect-server/src/prefect_server/services/towel/scheduler.py\", line 46, in run_once\n    offset=500 * iterations,\n  File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n    as_box=not apply_schema,\n  File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n    as_box=as_box,\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 84, in execute\n    raise ValueError(result[\"errors\"])\nValueError: [{'extensions': {'path': '$.selectionSet.flow', 'code': 'validation-failed'}, 'message': 'field \"flow\" not found in type: \\'query_root\\''}]"}
towel_1     | {"severity": "ERROR", "name": "prefect-server.ZombieKiller", "message": "Unexpected error: ValueError([{'extensions': {'path': '$.selectionSet.task_run', 'code': 'validation-failed'}, 'message': 'field \"task_run\" not found in type: \\'query_root\\''}])", "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n    await self.run_once()\n  File \"/prefect-server/src/prefect_server/services/towel/zombie_killer.py\", line 216, in run_once\n    await self.reap_zombie_task_runs()\n  File \"/prefect-server/src/prefect_server/services/towel/zombie_killer.py\", line 153, in reap_zombie_task_runs\n    apply_schema=False,\n  File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n    as_box=not apply_schema,\n  File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n    as_box=as_box,\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 84, in execute\n    raise ValueError(result[\"errors\"])\nValueError: [{'extensions': {'path': '$.selectionSet.task_run', 'code': 'validation-failed'}, 'message': 'field \"task_run\" not found in type: \\'query_root\\''}]"}
The UI doesn't connect either
😞
aswell, when running 
nmap localhost
shouldn't there be one for port 4200 ?
k

Kevin Kho

07/31/2021, 8:13 PM
I guess? Looks like the API didn’t spin up properly .what AMI are you using? i’ll give this a shot.
i

Ile Lee

07/31/2021, 8:18 PM
I've tried with Linux Ami 2
And then with Ubuntu
The current setup is with ubuntu
k

Kevin Kho

07/31/2021, 8:50 PM
So I tried this and stuff is starting. I can access the API from my local machine, but not the UI, so it seems like there might still be something wrong there. Gonna have to ask the team and get back to you on Monday, but 
towel
and the 
api
should at least start. I did this on t5.large Linux AMI 2. Just had to get docker and docker compose working and then 
prefect server start
.
i

Ile Lee

07/31/2021, 9:10 PM
I appreciate this so much, thank you.
k

Kevin Kho

08/02/2021, 3:08 PM
Hey @Ile Lee, I think the easiest thing to try is to use the 0.15.2 image like 
prefect server start --version core-0.15.2
i

Ile Lee

08/02/2021, 3:49 PM
@Kevin Kho Hi. Okay I shall try that. I've also tried locally on my pc, and somehow I can reach the UI, It just seems like a problem when trying on EC2
k

Kevin Kho

08/02/2021, 3:49 PM
you do reach it on 
localhost
or with an IP address?
i

Ile Lee

08/02/2021, 3:50 PM
On the computer I reached on localhost:8080
On the EC2 I used ip-address:8080
I've tried. 
prefect server start --version core-0.15.2
Doesn't work 😕 I've tried with other versions aswell. The closest I got was to get the UI's Navigation bar. I messed around with it and then it stopped working again... Will try again
k

Kevin Kho

08/02/2021, 5:40 PM
What do you mean with the UI Navigation bar? Does the API work (port 4200)?
i

Ile Lee

08/02/2021, 5:44 PM
I tried different versions using prefect server start --version <version-number> There was one version where I could see the blue bar on top with the Prefect Logo and some navigation links. But nothing else on the main page
k

Kevin Kho

08/02/2021, 5:51 PM
Hard to say what is going on. Honestly I would recommend a fresh start and using 
prefect server start --version core-0.15.2
. This should work. For you setup, it seems like some stuff is not working right. Do you have logs you can share? I used the Linux AMI 2 and t5.large when I got it working.
i

Ile Lee

08/03/2021, 11:38 AM
Ohhhh myyyyy goshhh!!! It works!!!
I started it from scratch and used a t3.large instance with Linux AMI 2 this time. Rather than a t2.micro. I wonder why though... Could it be that the instance was too small and low compute?
Thank you so much Kevin by the way for all your help and time !
k

Kevin Kho

08/03/2021, 1:38 PM
I think so. You probably didnt have enough resources to spin everything up. Thanks for the patience, and 0.15.3 was re tagged yesterday I think so it might work already.
i

Ile Lee

08/05/2021, 9:43 AM
Thank you for all the help!
#prefect-serverJoin Slack
Powered by