https://prefect.io logo
Join Slack
Powered by
I don’t know if this is a bug or not - I set a tas...
# ask-community
t
I don’t know if this is a bug or not - I set a task concurrency to 1 with one label defined and testing, however, it keeps hanging at state: 
Queued
with just one that task running. Only if I increase the limit to 2 it will start running. It looks a bug to me:
k
Hey @Tony Yun, am a bit confused. It seems there is a concurrency limit of 1 which is blocking another run from running? Do you have another task that is running? Or do you mean that no copy of the task is running and the usage should be 0?
t
I have another task running but without that label on. For this task in screenshot, it is the only one with this label running, however it’s getting queued.
k
Maybe you can try searching if there was a task that did not close successfully using the GraphQL API using something like this:
Copy code
query {
  task (where:{tags: {_contains: ["test"]}}){
    tags
    task_runs (where: {state: {_eq: "Running"}}){
      id
      name
      state
    }
  }
}
t
Ok I’ll take a look tomorrow. Thanks!
@Kevin Kho, only one task is running from running that query, but the task keeps getting queued with limit=1:
Copy code
{
  "data": {
    "task": [
      {
        "tags": [
          "dbt-run"
        ],
        "task_runs": []
      },
      {
        "tags": [
          "dbt-run"
        ],
        "task_runs": [
          {
            "id": "9f0d5227-635a-4a7c-bb1c-dbcb8cfe32f5",
            "name": null,
            "state": "Running"
          }
        ]
      }
    ]
  }
}
k
It seems like there is something stuck in a running state then somewhere? Maybe you can add the flow to the query and see if it’s the same one?
t
I see. Mind send me the query syntax?.. haven’t used graphQL much yet
k
Yeah one sec
Could you try this?
Copy code
query {
  task (where:{tags: {_contains: ["test"]}}){
    tags
    task_runs (where: {state: {_eq: "Running"}}){
      id
      name
      state
    }
    flow {
      flow_runs {
        id
        name
      }
  	}
  }
}
t
trying now
hmm, a lot returned:
Copy code
{
  "data": {
    "task": [
      {
        "tags": [
          "dbt-run"
        ],
        "task_runs": [],
        "flow": {
          "flow_runs": [
            {
              "id": "d347d2ac-4b59-4334-b00e-383064b052b4",
              "name": "proficient-swift"
            }
          ]
        }
      },
      {
        "tags": [
          "dbt-run"
        ],
        "task_runs": [
          {
            "id": "9f0d5227-635a-4a7c-bb1c-dbcb8cfe32f5",
            "name": null,
            "state": "Running"
          }
        ],
        "flow": {
          "flow_runs": [
            {
              "id": "2c170277-7143-4613-b3f0-13baed76c223",
              "name": "lumpy-cuttlefish"
            },
            {
              "id": "a775945e-4fbb-4df7-953b-c59a7c67c500",
              "name": "enthusiastic-waxbill"
            },
            {
              "id": "7e133c4a-652e-4f01-b960-a9f07418407b",
              "name": "brave-aardwolf"
            },
            {
              "id": "a8a32c83-e4e5-4cbd-a5d9-dab277e64fa2",
              "name": "airborne-goshawk"
            }
          ]
        }
      }
    ]
  }
}
don’t understand what’s going on. from looking at the Dashboard, only one flow is running at this moment.
there seems lot of ghost flows running that I don’t see on UI?
this 
airborne-goshawk
is the one I can only see on UI:
Copy code
{
  "id": "a8a32c83-e4e5-4cbd-a5d9-dab277e64fa2",
  "name": "airborne-goshawk"
}
k
Let me check with the team if my query is right
🙏 1
Ok this is the right form
Copy code
query {
  task (where:{tags: {_contains: ["test"]}}){
    tags
    task_runs (where: {state: {_eq: "Running"}}){
      id
      name
      state
      flow_run {
        flow {
          id
          name
        }
      }
    }
  }
}
The flow_run and flow need to be inside the task run
a
@Tony Yun how did you end up approaching the never ending flow problem? Do you have a state handler attached to this flow to ensure the flow run is triggered all over again? Can you share your Flow code? I think this way, we can identify the root cause easier, before even looking at concurrency limits
k
A flow concurrency limit might help you better than the task concurrency limit
upvote 1
t
ok running the new query, see it’s only returning one flow, which is expected:
Copy code
{
  "data": {
    "task": [
      {
        "tags": [
          "dbt-run"
        ],
        "task_runs": []
      },
      {
        "tags": [
          "dbt-run"
        ],
        "task_runs": [
          {
            "id": "9f0d5227-635a-4a7c-bb1c-dbcb8cfe32f5",
            "name": null,
            "state": "Running",
            "flow_run": {
              "flow": {
                "id": "1deaac33-4d51-4f69-b33a-751170044803",
                "name": "DBT flow"
              }
            }
          }
        ]
      }
    ]
  }
}
👍 1
but it’s still getting queued from one hour ago
i think task concurrency is pretty good enough for me to limit concurrency, flow concurrency required labels on agent. However our default agent has no any labels, to add labels will likely break all rest of our production flows which i don’t dare to give it a try.
k
Gotcha. Do you only have DBT task inside that flow? If the flow is not running, is the usage of the task tag 0/1?
t
yes, that’s correct. Only one task of 
RunK8sJob
. If I cancel it, it will be back to 0/1
k
Is your flow code simple enough to share for this flow?
t
ugh, not that simple. But what I know is, If I increase the limit to 2, it will get running. maybe becasue I refactored that 
Runk8sJob
class 😂..
inside the task, it’s calling another task 
ReadK8sLog
task. maybe it’s by default added label to that task, so only 
ReadLog
task is running. I think that’s maybe why it happened
k
Ok but to answer the original question, this is not normal behavior lol
t
I see I see. Make sense now!
thanks !
@Kevin Kho do you know to kill a ghost running task/flow given its id in GraphQL?
i see 3 flow runs in GraphQL but not any in dashbaord UI using your query:
Copy code
{
  "data": {
    "task": [
      {
        "tags": [
          "dbt-run"
        ],
        "task_runs": []
      },
      {
        "tags": [
          "dbt-run"
        ],
        "task_runs": [
          {
            "id": "93d54fa2-d2c0-42d7-9af9-f366c1dd76f9",
            "name": null,
            "state": "Running",
            "flow_run": {
              "flow": {
                "id": "1deaac33-4d51-4f69-b33a-751170044803",
                "name": "DBT flow"
              }
            }
          },
          {
            "id": "8bef4b09-dd24-4b58-a59f-33fbda30d8ad",
            "name": null,
            "state": "Running",
            "flow_run": {
              "flow": {
                "id": "1deaac33-4d51-4f69-b33a-751170044803",
                "name": "DBT flow"
              }
            }
          },
          {
            "id": "9f0d5227-635a-4a7c-bb1c-dbcb8cfe32f5",
            "name": null,
            "state": "Running",
            "flow_run": {
              "flow": {
                "id": "1deaac33-4d51-4f69-b33a-751170044803",
                "name": "DBT flow"
              }
            }
          }
        ]
      }
    ]
  }
}
k
You can set the state with the GraphQL example here: https://docs.prefect.io/orchestration/flow-runs/setting-states.html#graphql
👍 1
t
is there a python function/CLI command to stop a running task?
I tried that GraphQL, but the flow shows no found:
Copy code
{
  "data": {
    "flow_run_by_pk": null
  }
}
k
There is none in the CLI at the moment
Wait no not that haha. The two queries after. The mutation and the JSON for the state
t
ok let me try
Open in Slack
PreviousNext
Powered by