Thread
#prefect-server
    h

    Hugo Polloli

    1 year ago
    Hi, I'm trying to setup my prefect backend and one agent to be ever-running on my EC2 instance, docker is already automatically starting and since restart is set to true in docker-server's compose.yml, prefect server also automatically starts ever since I launched it once with --detach. But how do I make the agent automatically start on boot too ? according to the docs I could use supervisor with the local agent, is there any alternative using the docker agent ? Should I make my own compose.yml for the agent ?
    Michael Adkins

    Michael Adkins

    1 year ago
    @Hugo Polloli You can definitely add an agent container to the docker-compose file. You can run
    prefect server config
    to get a configured compose file then modify it to include an agent service or you can just have a separate compose file.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Hugo Polloli, I think you can also look into startup scripts.
    h

    Hugo Polloli

    1 year ago
    Hi thanks to you both ! I'll look into both of these solutions 🙂
    Ile Lee

    Ile Lee

    1 year ago
    @Hugo Polloli Hi Hugo. How did you get Prefect server to run on your EC2? I've tried Prefect backend server and prefect server start, but I cannot access the UI on the publicIP:8080
    Kevin Kho

    Kevin Kho

    1 year ago
    If you SSH into the EC2, can you see it on localhost:8080?
    h

    Hugo Polloli

    1 year ago
    If you try the above and can see on localhost:8080, then did you add a security group to your EC2 instance that opens both 8080 and 4200 ports to the public ?
    Ile Lee

    Ile Lee

    1 year ago
    hi @Hugo Polloli I have tried with opening ports 8080 and 4200 to the public. Funny thing is, on ubuntu it didnt work, but on linux it works. Maybe there's something I'm missing... Is there a post / thread / article you have followed to document the whole process from scratch ?
    h

    Hugo Polloli

    1 year ago
    Just in case, here're my inbound rules for the prefect security group I created,
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Ile Lee, I think what probably happened is the server containers did not start properly. Could you check the logs and see if everything is working? Or check the UI on localhost on the ubuntu server?
    Ile Lee

    Ile Lee

    1 year ago
    @Hugo Polloli @Kevin Kho I've tried again from scratch. There aren't many people who have deployed prefect server on AWS, and since you have, could you correct me if any of my steps are wrong? Sorry I'm still quite new to this field. This is what I've done in the N.Virginia region (us-east-1):Step 1: Created VPC • with CIDR block 10.100.0.0/16Step 2: Created Subnet • in us-east-1b • with CIDR block 10.100.1.0/24 • enabled auto assign public IPv4 addressStep 3: Created Internet Gateway • then attached it to the VPCStep 4: Created Route Table • Added destination 0.0.0.0 to target the internet gateway • In subnet association, added the subnet to explicit subnet associationStep 5: Created EC2 • Also created Security Group - added inbound rules TCP 8080 (for prefect UI), TCP 4200 (for graphql), TCP 22 (for SSH) to anyone (0.0.0.0/0) • Downloaded a new key pairStep 6: SSH into the EC2 instance • updated packages • installed docker, docker-compose, prefect (ran sudo service docker start, so docker ps works)Step 7: Created a directory in /home/ec2-user/.prefect Step 8 : Created a config.toml file in /home/ec2-user/.prefect with the following inside it
    [server]
           [server.ui]
           apollo_url = "<http://public-ip:4200/graphql>"
           graphql_url = "<http://public-ip:4200/graphql>"
    From there: prefect backend server prefect server start The server starts and says you can visit in localhost:8080 but when I go to my-public-ip:8080, the prefect UI doesn't show I'm sorry for such a long explanation, I just can't figure out why I can't get it running 😦
    I am using Linux AMI 2
    h

    Hugo Polloli

    1 year ago
    Did you apply the security group created at step 5 to the EC2 instance ? See screenshot, then add it. If you already have then sorry but I don't know what else to check 😕
    Ile Lee

    Ile Lee

    1 year ago
    I have 😞 Which AMI are you using?
    Kevin Kho

    Kevin Kho

    1 year ago
    Can you confirm localhost 4200 and 8080 are up when you ssh just so we know if it’s a networking issue or issue with spinning those up? Can you check both because the UI image had an issue and was fixed 2 days ago so someone reported 4200 working but now 8080
    Ile Lee

    Ile Lee

    1 year ago
    @Kevin Kho Hi. Sorry I wasn't sure exactly how to do it so I googled it and typed
    nmap localhost
    Output was as follows: PORT STATE SERVICE 22/tcp open ssh 3000/tcp open ppp 5432/tcp open postgresql 8080/tcp open http-proxy
    It doesn't seem to have port 4200
    Kevin Kho

    Kevin Kho

    1 year ago
    Maybe you can try hitting the API with
    prefect get flows
    ?
    Do you also have a browser you can use to open localhost:8080?
    Ile Lee

    Ile Lee

    1 year ago
    by localhost:8080 do you mean the Linux localhost?
    Because on my physical computer I go to the browser, type ec2-instance-public-ip:8080
    Kevin Kho

    Kevin Kho

    1 year ago
    Yep 8080 is the UI so I’m wondering if you can open the UI with a web browser on the EC2 instance so that we know if it’s a networking issue or the container didn’t spin up right. If you SSH into the instance,
    localhost:8080
    should show the browser. Btw, I should just mention because people don’t know this but Prefect Cloud has 10000 free task runs per month also, which is more than enough to get started.
    Ile Lee

    Ile Lee

    1 year ago
    So let's say I've SSH'ed into the EC2
    ran Prefect backend server, and prefect server start. lets say my public ec2 instance ip is 3.144.2.10 On firefox I go to 3.144.2.10:8080 This is what it says: Unable to connect Firefox can’t establish a connection to the server at 44.195.83.50:8080. The site could be temporarily unavailable or too busy. Try again in a few moments. If you are unable to load any pages, check your computer’s network connection. If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the Web.
    On the SSH terminal I do get the "Welcome to PREFECT SERVER" "Visit http://localhost:8080 to get started, or check the docs at https://docs.prefect.io
    I can't reach the UI for some reason
    Kevin Kho

    Kevin Kho

    1 year ago
    Can you try clearing the Docker images you have and then doing that again? There was a bug in the UI image fixed 2 days ago. If your UI image tagged as 0.15.3?
    Any logs in the UI container?
    Ile Lee

    Ile Lee

    1 year ago
    I see 0.15.3 !
    docker images shows core-0.15.3
    Kevin Kho

    Kevin Kho

    1 year ago
    Delete those images to force the re-download, and then do the
    prefect server start
    again since we fixed the image two days ago so you’ll be sure to pull the right image this time.
    Ile Lee

    Ile Lee

    1 year ago
    So I tried deleted the images using
    docker image prune -a
    then ran prefect server start but the images still have the 0.15.3
    Am I doing something wrong?
    Sorry I'm a bit of a noob
    Kevin Kho

    Kevin Kho

    1 year ago
    If you did the
    prefect server stop
    , I think prune should work to remove those images. The updates are still tagged as 0.15.3, they were updated. The important thing is you should see new images being downloaded when you do
    prefect server start
    Ile Lee

    Ile Lee

    1 year ago
    Awesome, it is pulling and downloading the images
    when i type docker images it still has the tag 0.15.3
    Kevin Kho

    Kevin Kho

    1 year ago
    That’s good. I would suggest watching the logs and monitoring if everything is starting successfully, if there’s no errors, you can try the UI at
    localhost:8080
    again, and hopefully it works. The UI image logs should also tell you if it started successfully. You can find that container with docker, and then get the logs for that specific container.
    Ile Lee

    Ile Lee

    1 year ago
    towel_1 is throwing errors. Prefect server Lazarus, prefect server Scheduler, prefect server zombie killer
    field not found in type query_root
    Kevin Kho

    Kevin Kho

    1 year ago
    Could you post your error message here? I think that the UI will at least be up 😅
    Ile Lee

    Ile Lee

    1 year ago
    towel_1     | {"severity": "ERROR", "name": "prefect-server.Lazarus", "message": "Unexpected error: ValueError([{'extensions': {'path': '$.selectionSet.flow_run', 'code': 'validation-failed'}, 'message': 'field \"flow_run\" not found in type: \\'query_root\\''}])", "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n    await self.run_once()\n  File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 37, in run_once\n    return await self.reschedule_flow_runs()\n  File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 89, in reschedule_flow_runs\n    limit=5000,\n  File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n    as_box=not apply_schema,\n  File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n    as_box=as_box,\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 84, in execute\n    raise ValueError(result[\"errors\"])\nValueError: [{'extensions': {'path': '$.selectionSet.flow_run', 'code': 'validation-failed'}, 'message': 'field \"flow_run\" not found in type: \\'query_root\\''}]"}
    towel_1     | {"severity": "ERROR", "name": "prefect-server.Scheduler", "message": "Unexpected error: ValueError([{'extensions': {'path': '$.selectionSet.flow', 'code': 'validation-failed'}, 'message': 'field \"flow\" not found in type: \\'query_root\\''}])", "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n    await self.run_once()\n  File \"/prefect-server/src/prefect_server/services/towel/scheduler.py\", line 46, in run_once\n    offset=500 * iterations,\n  File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n    as_box=not apply_schema,\n  File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n    as_box=as_box,\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 84, in execute\n    raise ValueError(result[\"errors\"])\nValueError: [{'extensions': {'path': '$.selectionSet.flow', 'code': 'validation-failed'}, 'message': 'field \"flow\" not found in type: \\'query_root\\''}]"}
    towel_1     | {"severity": "ERROR", "name": "prefect-server.ZombieKiller", "message": "Unexpected error: ValueError([{'extensions': {'path': '$.selectionSet.task_run', 'code': 'validation-failed'}, 'message': 'field \"task_run\" not found in type: \\'query_root\\''}])", "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n    await self.run_once()\n  File \"/prefect-server/src/prefect_server/services/towel/zombie_killer.py\", line 216, in run_once\n    await self.reap_zombie_task_runs()\n  File \"/prefect-server/src/prefect_server/services/towel/zombie_killer.py\", line 153, in reap_zombie_task_runs\n    apply_schema=False,\n  File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n    as_box=not apply_schema,\n  File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n    as_box=as_box,\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 84, in execute\n    raise ValueError(result[\"errors\"])\nValueError: [{'extensions': {'path': '$.selectionSet.task_run', 'code': 'validation-failed'}, 'message': 'field \"task_run\" not found in type: \\'query_root\\''}]"}
    The UI doesn't connect either
    😞
    aswell, when running
    nmap localhost
    shouldn't there be one for port 4200 ?
    Kevin Kho

    Kevin Kho

    1 year ago
    I guess? Looks like the API didn’t spin up properly .what AMI are you using? i’ll give this a shot.
    Ile Lee

    Ile Lee

    1 year ago
    I've tried with Linux Ami 2
    And then with Ubuntu
    The current setup is with ubuntu
    Kevin Kho

    Kevin Kho

    1 year ago
    So I tried this and stuff is starting. I can access the API from my local machine, but not the UI, so it seems like there might still be something wrong there. Gonna have to ask the team and get back to you on Monday, but
    towel
    and the
    api
    should at least start. I did this on t5.large Linux AMI 2. Just had to get docker and docker compose working and then
    prefect server start
    .
    Ile Lee

    Ile Lee

    1 year ago
    I appreciate this so much, thank you.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Ile Lee, I think the easiest thing to try is to use the 0.15.2 image like
    prefect server start --version core-0.15.2
    Ile Lee

    Ile Lee

    1 year ago
    @Kevin Kho Hi. Okay I shall try that. I've also tried locally on my pc, and somehow I can reach the UI, It just seems like a problem when trying on EC2
    Kevin Kho

    Kevin Kho

    1 year ago
    you do reach it on
    localhost
    or with an IP address?
    Ile Lee

    Ile Lee

    1 year ago
    On the computer I reached on localhost:8080
    On the EC2 I used ip-address:8080
    I've tried.
    prefect server start --version core-0.15.2
    Doesn't work 😕 I've tried with other versions aswell. The closest I got was to get the UI's Navigation bar. I messed around with it and then it stopped working again... Will try again
    Kevin Kho

    Kevin Kho

    1 year ago
    What do you mean with the UI Navigation bar? Does the API work (port 4200)?
    Ile Lee

    Ile Lee

    1 year ago
    I tried different versions using prefect server start --version <version-number> There was one version where I could see the blue bar on top with the Prefect Logo and some navigation links. But nothing else on the main page
    Kevin Kho

    Kevin Kho

    1 year ago
    Hard to say what is going on. Honestly I would recommend a fresh start and using
    prefect server start --version core-0.15.2
    . This should work. For you setup, it seems like some stuff is not working right. Do you have logs you can share? I used the Linux AMI 2 and t5.large when I got it working.
    Ile Lee

    Ile Lee

    1 year ago
    Ohhhh myyyyy goshhh!!! It works!!!
    I started it from scratch and used a t3.large instance with Linux AMI 2 this time. Rather than a t2.micro. I wonder why though... Could it be that the instance was too small and low compute?
    Thank you so much Kevin by the way for all your help and time !
    Kevin Kho

    Kevin Kho

    1 year ago
    I think so. You probably didnt have enough resources to spin everything up. Thanks for the patience, and 0.15.3 was re tagged yesterday I think so it might work already.
    Ile Lee

    Ile Lee

    1 year ago
    Thank you for all the help!