a

    Arthur Jacquemart

    3 months ago
    Hi Prefect community. I am playing around with the GraphQL API and I do not understand the output. I want to check if an agent has some task "pending" or "running". I managed to do it but I do not understand the answer I get back from the API
    Anna Geller

    Anna Geller

    3 months ago
    could you move the code blocks to the thread?
    a

    Arthur Jacquemart

    3 months ago
    Here is the query:
    {
      task_run(
        where: {state: {_in: ["Running", "Pending"]}, flow_run: {labels: {_eq: ["scaled"]}}}
      ) {
        state
        flow_run {
          flow {
            name
          }
          labels
        }
      }
    }
    Here is an excerpt of the answer:
    {
      "data": {
        "task_run": [
          {
            "flow_run": {
              "name": "affable-butterfly",
              "flow": {
                "name": "whatever"
              },
              "labels": [
                "scaled"
              ]
            }
          },
          {
            "flow_run": {
              "name": "affable-butterfly",
              "flow": {
                "name": "whatever"
              },
              "labels": [
                "scaled"
              ]
            }
          },
          {
            "flow_run": {
              "name": "affable-butterfly",
              "flow": {
                "name": "whatever"
              },
              "labels": [
                "scaled"
              ]
            }
          },
    Which looks good at first sight. But this flow is not pending: ā€¢ I cannot see it in the UI ā€¢ It has no schedule attached to it I also dont understand why it returns duplicates I have the same behaviour with a flow showing as "running" in the GraphQL answer but which does not exist. Would you know what is causing this? Thanks !
    Anna Geller

    Anna Geller

    3 months ago
    your query asks about task runs in a pending state, not flow runs Those are not duplicates but various state types - you can add more fields to your query to validate that Can you explain a bit what problem are you trying to solve? you can explore the states directly from the UI
    UI makes it much easier to browse and explore the states and will be less confusing
    a

    Arthur Jacquemart

    3 months ago
    Ok working fine with "flow_run" instead of "task_run" I am trying to check if a particular agent has some flows schedules, running, or pending. The end goal is to create a flow to delete the agent if nothing is running/scheduled/pending.
    Anna Geller

    Anna Geller

    3 months ago
    Maybe a serverless agent is what you are looking for? check out AWS Fargate, both EKS on Fargate (recommended) or ECS Fargate (hard to work with API)
    agents are really meant to be on 24/7 and they are meant to be used as lightweight processes - your actual execution of a flow run can take place somewhere else e.g. on a remote Kubernetes cluster
    a

    Arthur Jacquemart

    3 months ago
    We have an azure stack so we cannot use AWS products... We run our agents in a k8s cluster hosted in azure. The problem is that the agent needs to live in the same node than where the flows will be run. So if a flow needs high memory/cpu requirements to be run, the node needs to be up and running all day long (as it must host the agent). This is what I understood so far. So one workaround would be to delete the agent running in the "powerful node" when we dont need it anymore. So the powerful node pool would scale down to 0 node, and we wont pay for the expensive compute when no jobs are running on it. do you believe there is a better solution using Azure clouds products and integrations with prefect?
    Anna Geller

    Anna Geller

    3 months ago
    you can use a Kubernetes agent and this way you can offload that resource management to Kubernetes instead of having to build the process for it yourself
    Azure has AKS
    a

    Arthur Jacquemart

    3 months ago
    yes that is what we are using, but i thought the kubernetes agent needed to run in the same node where the jobs will be running
    Anna Geller

    Anna Geller

    3 months ago
    You are thinking about an agent as a "big" thing that executes everything. Think about the agent as a tiny, lightweight process, only polling for runs and delegating work to someone else (e.g. K8s cluster) and K8s allows taints and tolerations if some work needs to occur on a specific node - do you have some infrastructure engineer/admin in your team you could ask for help here? this is a DevOps problem
    a

    Arthur Jacquemart

    3 months ago
    Not really, as i said in this message, i would love to have the agent running in a lightweight node, distributing jobs to other more powerful node but it looked like it was not possible. https://prefect-community.slack.com/archives/CL09KU1K7/p1654089295297919?thread_ts=1654084025.843109&cid=CL09KU1K7
    Anna Geller

    Anna Geller

    3 months ago
    oh I'm sorry now I understand what you mean
    do you have an option to schedule your flows during the day, say between 9 to 5 and then shut down this expensive VM automatically every day say at 6 pm?
    this is something that might work fine with some automation from your cloud provider doing instance scheduling and running agent process with supervisor and reboot crontab
    I'm not sure if I'm the right person to ask here - are you a Prefect Cloud customer already? we have paid support with experts who can help with infrastructure setup - check this
    a

    Arthur Jacquemart

    3 months ago
    no i cannot, because the flow is triggered when a blob storage receives a new csv. so it is quite unpredictable as it depends when our vendors deliver the data. I have already a flow which checks if a new file has been uploaded, so i could start the VM after that, and then add a flow to stop the VM at the end of my heavy flow, given no other flows are running/pending on it (hence my question about the graphQL)
    yes we are prefect cloud customers
    Anna Geller

    Anna Geller

    3 months ago
    Great, I think in that case, you can ask cs@prefect.io for help here since this is a bit challenging to give help via community support - there's a lot of complexity here, e.g., if you schedule a flow to stop the agent, you would ask the agent to commit suicide šŸ˜„ it's not possible this way and requires a longer conversation for which we have infrastructure experts in the CS team
    a

    Arthur Jacquemart

    3 months ago
    Thanks Anna, I booked a call tomorrow with Allie Thank you again for your help. Sorry if i was not clear in my messages
    Anna Geller

    Anna Geller

    3 months ago
    Amazing! all good, you are in good hands now šŸ™Œ