https://prefect.io logo
Powered by
Title
t

Tony Yun

06/14/2022, 6:26 PM
Hi, I think there is a ghost task running behind which we cannot see on UI. All of our running task is queued but no job is seen running on UI. Can someone help? 
Task 'RunNamespacedJob - DBT Run': Finished task run for task with final state: 'Queued'
I have some trouble in making a GraphQL to find that ghost running task out and neither I know how to kill it if I found it out. I’ve checked this configuration which says no task is running:
k

Kevin Kho

06/14/2022, 6:32 PM
Does it actually run if you remove the limit or make it 2? I don’t expect it to?
t

Tony Yun

06/14/2022, 6:33 PM
no, even after making it 2
actually I made the graphQL query, it is the only flow running at this moment:
k

Kevin Kho

06/14/2022, 6:36 PM
I don’t think this is a concurrency issue. The Flow Run concurrency page should be accurate because we had a bug that we fixed a couple of months back. If your GraphQL Query also returns that, then I think it’s right. This looks like something got stuck on the agent side. Are there more agent logs you can look at?
t

Tony Yun

06/14/2022, 6:37 PM
where can I see more agent logs? Nothing more I see than this task is getting Queued
ok I think I found it in GraphQL
{
        "id": "f0f2a6fb-c2b6-4cb4-9fd8-e3ea7f7becf4",
        "state": "Running",
        "start_time": "2022-06-14T17:40:52.663061+00:00",
        "name": null,
        "flow_run": {
          "id": "918b4564-91bf-40fd-ac53-7b6f96d855ad",
          "name": "large-teal"
        },
        "flow_run_id": "918b4564-91bf-40fd-ac53-7b6f96d855ad",
        "task": {
          "id": "df9c3284-df86-453f-ad38-cf43c9cc22f5",
          "name": "RunNamespacedJob - DBT Run"
        }
      },
how can I kill it?
k

Kevin Kho

06/14/2022, 6:44 PM
You can mark it as failed or cancelled in the UI (preferably Failed) and that should kill the process too because it will error out the next time it tries to update state
t

Tony Yun

06/14/2022, 6:44 PM
no I don’t see it on UI
that’s the main problem
k

Kevin Kho

06/14/2022, 6:45 PM
what happens if you navigate to that page using that 
flow_run_id
?
t

Tony Yun

06/14/2022, 6:45 PM
let me try
that worked!
it’s flow state is Canceled though?
the flow-run name is matched 
large-teal
k

Kevin Kho

06/14/2022, 6:46 PM
Is there a task in it still running? maybe mark those as failed
t

Tony Yun

06/14/2022, 6:47 PM
oh yeah
saw it
it’s a weird bug:
killed. everything all good now. Thanks @Kevin Kho!
👍 1
4 Views
#prefect-communityJoin Slack
Powered by