Hi Prefect community. I am playing around with the GraphQL API and I do not understand the output. I want to check if an agent has some task "pending" or "running". I managed to do it but I do not understand the answer I get back from the API

could you move the code blocks to the thread?

Here is the query:

{
  task_run(
    where: {state: {_in: ["Running", "Pending"]}, flow_run: {labels: {_eq: ["scaled"]}}}
  ) {
    state
    flow_run {
      flow {
        name
      }
      labels
    }
  }
}

Here is an excerpt of the answer:

{
  "data": {
    "task_run": [
      {
        "flow_run": {
          "name": "affable-butterfly",
          "flow": {
            "name": "whatever"
          },
          "labels": [
            "scaled"
          ]
        }
      },
      {
        "flow_run": {
          "name": "affable-butterfly",
          "flow": {
            "name": "whatever"
          },
          "labels": [
            "scaled"
          ]
        }
      },
      {
        "flow_run": {
          "name": "affable-butterfly",
          "flow": {
            "name": "whatever"
          },
          "labels": [
            "scaled"
          ]
        }
      },

Which looks good at first sight. But this flow is not pending: • I cannot see it in the UI • It has no schedule attached to it I also dont understand why it returns duplicates I have the same behaviour with a flow showing as "running" in the GraphQL answer but which does not exist. Would you know what is causing this? Thanks !

your query asks about task runs in a pending state, not flow runs Those are not duplicates but various state types - you can add more fields to your query to validate that Can you explain a bit what problem are you trying to solve? you can explore the states directly from the UI

Ok working fine with "flow_run" instead of "task_run" I am trying to check if a particular agent has some flows schedules, running, or pending. The end goal is to create a flow to delete the agent if nothing is running/scheduled/pending.

Maybe a serverless agent is what you are looking for? check out AWS Fargate, both EKS on Fargate (recommended) or ECS Fargate (hard to work with API)

We have an azure stack so we cannot use AWS products... We run our agents in a k8s cluster hosted in azure. The problem is that the agent needs to live in the same node than where the flows will be run. So if a flow needs high memory/cpu requirements to be run, the node needs to be up and running all day long (as it must host the agent). This is what I understood so far. So one workaround would be to delete the agent running in the "powerful node" when we dont need it anymore. So the powerful node pool would scale down to 0 node, and we wont pay for the expensive compute when no jobs are running on it. do you believe there is a better solution using Azure clouds products and integrations with prefect?

you can use a Kubernetes agent and this way you can offload that resource management to Kubernetes instead of having to build the process for it yourself

yes that is what we are using, but i thought the kubernetes agent needed to run in the same node where the jobs will be running

You are thinking about an agent as a "big" thing that executes everything. Think about the agent as a tiny, lightweight process, only polling for runs and delegating work to someone else (e.g. K8s cluster) and K8s allows taints and tolerations if some work needs to occur on a specific node - do you have some infrastructure engineer/admin in your team you could ask for help here? this is a DevOps problem

Not really, as i said in this message, i would love to have the agent running in a lightweight node, distributing jobs to other more powerful node but it looked like it was not possible. https://prefect-community.slack.com/archives/CL09KU1K7/p1654089295297919?thread_ts=1654084025.843109&cid=CL09KU1K7

oh I'm sorry now I understand what you mean

no i cannot, because the flow is triggered when a blob storage receives a new csv. so it is quite unpredictable as it depends when our vendors deliver the data. I have already a flow which checks if a new file has been uploaded, so i could start the VM after that, and then add a flow to stop the VM at the end of my heavy flow, given no other flows are running/pending on it (hence my question about the graphQL)

Great, I think in that case, you can ask cs@prefect.io for help here since this is a bit challenging to give help via community support - there's a lot of complexity here, e.g., if you schedule a flow to stop the agent, you would ask the agent to commit suicide 😄 it's not possible this way and requires a longer conversation for which we have infrastructure experts in the CS team

Thanks Anna, I booked a call tomorrow with Allie Thank you again for your help. Sorry if i was not clear in my messages

Amazing! all good, you are in good hands now 🙌