Tony Yun

    Tony Yun

    3 months ago
    Hi, I think there is a ghost task running behind which we cannot see on UI. All of our running task is queued but no job is seen running on UI. Can someone help?
    Task 'RunNamespacedJob - DBT Run': Finished task run for task with final state: 'Queued'
    I have some trouble in making a GraphQL to find that ghost running task out and neither I know how to kill it if I found it out. I’ve checked this configuration which says no task is running:
    Kevin Kho

    Kevin Kho

    3 months ago
    Does it actually run if you remove the limit or make it 2? I don’t expect it to?
    Tony Yun

    Tony Yun

    3 months ago
    no, even after making it 2
    actually I made the graphQL query, it is the only flow running at this moment:
    Kevin Kho

    Kevin Kho

    3 months ago
    I don’t think this is a concurrency issue. The Flow Run concurrency page should be accurate because we had a bug that we fixed a couple of months back. If your GraphQL Query also returns that, then I think it’s right. This looks like something got stuck on the agent side. Are there more agent logs you can look at?
    Tony Yun

    Tony Yun

    3 months ago
    where can I see more agent logs? Nothing more I see than this task is getting Queued
    ok I think I found it in GraphQL
    {
            "id": "f0f2a6fb-c2b6-4cb4-9fd8-e3ea7f7becf4",
            "state": "Running",
            "start_time": "2022-06-14T17:40:52.663061+00:00",
            "name": null,
            "flow_run": {
              "id": "918b4564-91bf-40fd-ac53-7b6f96d855ad",
              "name": "large-teal"
            },
            "flow_run_id": "918b4564-91bf-40fd-ac53-7b6f96d855ad",
            "task": {
              "id": "df9c3284-df86-453f-ad38-cf43c9cc22f5",
              "name": "RunNamespacedJob - DBT Run"
            }
          },
    how can I kill it?
    Kevin Kho

    Kevin Kho

    3 months ago
    You can mark it as failed or cancelled in the UI (preferably Failed) and that should kill the process too because it will error out the next time it tries to update state
    Tony Yun

    Tony Yun

    3 months ago
    no I don’t see it on UI
    that’s the main problem
    Kevin Kho

    Kevin Kho

    3 months ago
    what happens if you navigate to that page using that
    flow_run_id
    ?
    Tony Yun

    Tony Yun

    3 months ago
    let me try
    that worked!
    it’s flow state is Canceled though?
    the flow-run name is matched
    large-teal
    Kevin Kho

    Kevin Kho

    3 months ago
    Is there a task in it still running? maybe mark those as failed
    Tony Yun

    Tony Yun

    3 months ago
    oh yeah
    saw it
    it’s a weird bug:
    killed. everything all good now. Thanks @Kevin Kho!