https://prefect.io logo
Title
a

Arthur Jacquemart

06/08/2022, 11:26 AM
Hi Prefect community. I am playing around with the GraphQL API and I do not understand the output. I want to check if an agent has some task "pending" or "running". I managed to do it but I do not understand the answer I get back from the API
āœ… 1
a

Anna Geller

06/08/2022, 11:45 AM
could you move the code blocks to the thread?
šŸ‘ 1
a

Arthur Jacquemart

06/08/2022, 11:50 AM
Here is the query:
{
  task_run(
    where: {state: {_in: ["Running", "Pending"]}, flow_run: {labels: {_eq: ["scaled"]}}}
  ) {
    state
    flow_run {
      flow {
        name
      }
      labels
    }
  }
}
Here is an excerpt of the answer:
{
  "data": {
    "task_run": [
      {
        "flow_run": {
          "name": "affable-butterfly",
          "flow": {
            "name": "whatever"
          },
          "labels": [
            "scaled"
          ]
        }
      },
      {
        "flow_run": {
          "name": "affable-butterfly",
          "flow": {
            "name": "whatever"
          },
          "labels": [
            "scaled"
          ]
        }
      },
      {
        "flow_run": {
          "name": "affable-butterfly",
          "flow": {
            "name": "whatever"
          },
          "labels": [
            "scaled"
          ]
        }
      },
Which looks good at first sight. But this flow is not pending: ā€¢ I cannot see it in the UI ā€¢ It has no schedule attached to it I also dont understand why it returns duplicates I have the same behaviour with a flow showing as "running" in the GraphQL answer but which does not exist. Would you know what is causing this? Thanks !
a

Anna Geller

06/08/2022, 11:51 AM
your query asks about task runs in a pending state, not flow runs Those are not duplicates but various state types - you can add more fields to your query to validate that Can you explain a bit what problem are you trying to solve? you can explore the states directly from the UI
šŸ‘Œ 1
UI makes it much easier to browse and explore the states and will be less confusing
a

Arthur Jacquemart

06/08/2022, 12:04 PM
Ok working fine with "flow_run" instead of "task_run" I am trying to check if a particular agent has some flows schedules, running, or pending. The end goal is to create a flow to delete the agent if nothing is running/scheduled/pending.
a

Anna Geller

06/08/2022, 12:11 PM
Maybe a serverless agent is what you are looking for? check out AWS Fargate, both EKS on Fargate (recommended) or ECS Fargate (hard to work with API)
agents are really meant to be on 24/7 and they are meant to be used as lightweight processes - your actual execution of a flow run can take place somewhere else e.g. on a remote Kubernetes cluster
a

Arthur Jacquemart

06/08/2022, 12:25 PM
We have an azure stack so we cannot use AWS products... We run our agents in a k8s cluster hosted in azure. The problem is that the agent needs to live in the same node than where the flows will be run. So if a flow needs high memory/cpu requirements to be run, the node needs to be up and running all day long (as it must host the agent). This is what I understood so far. So one workaround would be to delete the agent running in the "powerful node" when we dont need it anymore. So the powerful node pool would scale down to 0 node, and we wont pay for the expensive compute when no jobs are running on it. do you believe there is a better solution using Azure clouds products and integrations with prefect?
a

Anna Geller

06/08/2022, 12:27 PM
you can use a Kubernetes agent and this way you can offload that resource management to Kubernetes instead of having to build the process for it yourself
Azure has AKS
a

Arthur Jacquemart

06/08/2022, 12:29 PM
yes that is what we are using, but i thought the kubernetes agent needed to run in the same node where the jobs will be running
a

Anna Geller

06/08/2022, 12:35 PM
You are thinking about an agent as a "big" thing that executes everything. Think about the agent as a tiny, lightweight process, only polling for runs and delegating work to someone else (e.g. K8s cluster) and K8s allows taints and tolerations if some work needs to occur on a specific node - do you have some infrastructure engineer/admin in your team you could ask for help here? this is a DevOps problem
a

Arthur Jacquemart

06/08/2022, 12:39 PM
Not really, as i said in this message, i would love to have the agent running in a lightweight node, distributing jobs to other more powerful node but it looked like it was not possible. https://prefect-community.slack.com/archives/CL09KU1K7/p1654089295297919?thread_ts=1654084025.843109&cid=CL09KU1K7
a

Anna Geller

06/08/2022, 12:41 PM
oh I'm sorry now I understand what you mean
do you have an option to schedule your flows during the day, say between 9 to 5 and then shut down this expensive VM automatically every day say at 6 pm?
this is something that might work fine with some automation from your cloud provider doing instance scheduling and running agent process with supervisor and reboot crontab
I'm not sure if I'm the right person to ask here - are you a Prefect Cloud customer already? we have paid support with experts who can help with infrastructure setup - check this
a

Arthur Jacquemart

06/08/2022, 12:47 PM
no i cannot, because the flow is triggered when a blob storage receives a new csv. so it is quite unpredictable as it depends when our vendors deliver the data. I have already a flow which checks if a new file has been uploaded, so i could start the VM after that, and then add a flow to stop the VM at the end of my heavy flow, given no other flows are running/pending on it (hence my question about the graphQL)
yes we are prefect cloud customers
a

Anna Geller

06/08/2022, 1:02 PM
Great, I think in that case, you can ask cs@prefect.io for help here since this is a bit challenging to give help via community support - there's a lot of complexity here, e.g., if you schedule a flow to stop the agent, you would ask the agent to commit suicide šŸ˜„ it's not possible this way and requires a longer conversation for which we have infrastructure experts in the CS team
a

Arthur Jacquemart

06/08/2022, 1:05 PM
Thanks Anna, I booked a call tomorrow with Allie Thank you again for your help. Sorry if i was not clear in my messages
a

Anna Geller

06/08/2022, 1:07 PM
Amazing! all good, you are in good hands now šŸ™Œ