Thread
#prefect-community
    Gonzalo

    Gonzalo

    1 year ago
    Hello everyone 🙌 I'm having a problem with scheduling flows, all of my flows did not schedule automatically overnight and I can't activate them through the interactive API (When I try to
    set_schedule_active
    I always get
    "success": false
    ). When I try to manually toggle the flow through the GUI the entire server hangs until I manually kill the GraphQL process on the server. Can anyone help or provide some insight? I'm acting as an interpreter for the tech team so I have to do some back and forth with my questions.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Gonzalo, that’s weird. It seems to be performance related on the GraphQL pod? This is Prefect Server right? Are all of your pods healthy? Is the db running out of memory?
    Gonzalo

    Gonzalo

    1 year ago
    Hey Kevin, yes, it is indeed Prefect Server. We're using Fargate Agent, not Kubernetes, so we're not sure what to answer regarding pods. The tech team says they're not sure what you mean when you refer to a db running out of memory.
    I just recieved this error message from them
    Zach Angell

    Zach Angell

    1 year ago
    @Gonzalo it looks like they're trying to start Prefect Server but port 5432 is already allocated, so Postgres is unable to start
    If Prefect Server isn't running properly, you will not be able to edit schedules
    Gonzalo

    Gonzalo

    1 year ago
    Understood, I'll relay the information and I hope that solves it, thanks for the heads up!
    They're still having trouble. They restarted Prefect entirely, the following error occurred upon running and none of the agents are running right now
    Kevin Kho

    Kevin Kho

    1 year ago
    The agent not spinning up makes sense if server doesn’t spin up. Did all of the containers spin up healthy when you did
    prefect server start
    ? Still seems like something may have not spin up right
    Gonzalo

    Gonzalo

    1 year ago
    They're checking the logs right now
    All the containers seem to be ok, this is the latest error
    Kevin Kho

    Kevin Kho

    1 year ago
    This is when starting an agent right? Are you able to access the UI/dashboard?
    Gonzalo

    Gonzalo

    1 year ago
    Yes, and yes I can access the dashboard
    Kevin Kho

    Kevin Kho

    1 year ago
    Can you try
    query { hello }
    in the interactive API?
    Gonzalo

    Gonzalo

    1 year ago
    It returns
    {
      "data": {
        "hello": "👋"
      }
    }
    Kevin Kho

    Kevin Kho

    1 year ago
    And when you query for flows, you see stuff?
    query {flow {
      id
    }}
    Gonzalo

    Gonzalo

    1 year ago
    I see several IDs
    Kevin Kho

    Kevin Kho

    1 year ago
    What kind of agent are you starting? Is that a docker agent?
    Gonzalo

    Gonzalo

    1 year ago
    It's a Fargate agent
    Kevin Kho

    Kevin Kho

    1 year ago
    Can you show me your config toml? just omit sensitive info
    Gonzalo

    Gonzalo

    1 year ago
    One second
    [server]
      endpoint = "http://<name>:4200"
      request_timeout = 60
      [server.ui]
        apollo_url = "http://<name>:4200/graphql"
    where "<name>" is the url
    Kevin Kho

    Kevin Kho

    1 year ago
    Can you try adding
    /graphql
    after endpoint?
    Is this the config.toml of the server or agent? I meant for the agent
    Gonzalo

    Gonzalo

    1 year ago
    That's the server's file, they're telling me the agent has no .toml
    Kevin Kho

    Kevin Kho

    1 year ago
    Is the server on the same machine as the agent?
    Gonzalo

    Gonzalo

    1 year ago
    The server is on an EC2 instance and the agent is on an ECS machine
    (I'm deeply sorry for this cumbersome game of telephone, but I'm also very grateful for your time)
    Kevin Kho

    Kevin Kho

    1 year ago
    So with this walkthrough, you want
    [server]
      [server.ui]
        apollo_url = "<http://YOUR_MACHINES_PUBLIC_IP:4200/graphql>"
    on the server and
    [server]
    endpoint = "YOUR_MACHINES_PUBLIC_IP:4200/graphql"
    for the agent. and also
    prefect backend server
    of course. Really weird if your agent has no config.toml cuz it looks like it’s hitting a nearly correct endpoint there.
    And of course! no worries
    Gonzalo

    Gonzalo

    1 year ago
    I think we're not really understanding the config.toml as far as the agent goes, we only have one for the server (we tried adding "/graphql" to the end of the endpoint url to no avail).
    We have a backend.toml which reads:
    backend = "server"
    that's on the same directory
    Kevin Kho

    Kevin Kho

    1 year ago
    You can make a
    config.toml
    for the agent also in that same
    .prefect
    directory
    Gonzalo

    Gonzalo

    1 year ago
    I think we found something, is GraphQL all on port 4200 by default? Some of our environment variables (such as GRAPHQL_SERVICE_PORT) point to 4201 instead
    Also, making a
    config.toml
    for the agent under the same directory is confusing since we already have a file with that same name
    Kevin Kho

    Kevin Kho

    1 year ago
    Not inside your working directory. Inside the
    .prefect
    directory where the
    backend.toml
    is? You already have a config? GraphQL on port 4201 looks good. But your endpoint will be on 4200
    They are separate here
    4200 is apollo, 4201 is the graphql
    Gonzalo

    Gonzalo

    1 year ago
    Yeah, there's already a
    config.toml
    there
    Zach Angell

    Zach Angell

    1 year ago
    @Gonzalo are you able to edit the existing
    .prefect/config.toml
    file on the agent ECS machine to include Kevin's addition?
    [server]
    endpoint = "YOUR_MACHINES_PUBLIC_IP:4200/graphql"
    If you do have access to an interactive session on the agent machine, can you also confirm the machine can ping the server? This would rule out AWS networking issues that might be blocking the connection