Channels
#prefect-community
Title
z
Zachary Loertscher
08/15/2023, 2:17 PMHi all - I have a prefect job that kicks off airbyte connections. However, after creating all the tasks, they all hang in "pending"
These airbyte connections sit on an EC2 - I've rebooted it a couple of times to no avail. Prefect agent is healthy. Including logs in thread - any ideas?
Is there something I need to re-set on the prefect side? I had some flows yesterday that got stuck in "cancelling" that I deleted - could that be causing my tasks to get stuck in pending?
Please help!
Prefect version - 2.8.7
✅ 1
Copy code
Downloading flow code from storage at ''
10:14:39 AM
prefect.flow_runs
Created task run '{dev_stage} - fetch airbyte ConnectionIds-0' for task '{dev_stage} - fetch airbyte ConnectionIds'
10:14:40 AM
prefect.flow_runs
Executing '{dev_stage} - fetch airbyte ConnectionIds-0' immediately...
10:14:40 AM
prefect.flow_runs
fetch airbyte connection id's for the prd environment that run on a daily basis
10:14:40 AM
{dev_stage} - fetch airbyte ConnectionIds-0
prefect.task_runs
Finished in state Completed()
10:14:40 AM
{dev_stage} - fetch airbyte ConnectionIds-0
prefect.task_runs
Created task run 'create airbyte connections list-0' for task 'create airbyte connections list'
10:14:40 AM
prefect.flow_runs
Executing 'create airbyte connections list-0' immediately...
10:14:40 AM
prefect.flow_runs
converting airbyte connectionids into a dictionary
10:14:41 AM
create airbyte connections list-0
prefect.task_runs
Finished in state Completed()
10:14:41 AM
create airbyte connections list-0
prefect.task_runs
[PrefectFuture('prd - Airbyte - Task-0'), PrefectFuture('prd - Airbyte - Task-1'), PrefectFuture('prd - Airbyte - Task-2'), PrefectFuture('prd - Airbyte - Task-3'), PrefectFuture('prd - Airbyte - Task-4'), PrefectFuture('prd - Airbyte - Task-5'), PrefectFuture('prd - Airbyte - Task-6'), PrefectFuture('prd - Airbyte - Task-7'), PrefectFuture('prd - Airbyte - Task-8'), PrefectFuture('prd - Airbyte - Task-9'), PrefectFuture('prd - Airbyte - Task-10'), PrefectFuture('prd - Airbyte - Task-11'), PrefectFuture('prd - Airbyte - Task-12'), PrefectFuture('prd - Airbyte - Task-13'), PrefectFuture('prd - Airbyte - Task-14'), PrefectFuture('prd - Airbyte - Task-15'), PrefectFuture('prd - Airbyte - Task-16'), PrefectFuture('prd - Airbyte - Task-17'), PrefectFuture('prd - Airbyte - Task-18'), PrefectFuture('prd - Airbyte - Task-19'), PrefectFuture('prd - Airbyte - Task-20'), PrefectFuture('prd - Airbyte - Task-21'), PrefectFuture('prd - Airbyte - Task-22'), PrefectFuture('prd - Airbyte - Task-23'), PrefectFuture('prd - Airbyte - Task-24'), PrefectFuture('prd - Airbyte - Task-25'), PrefectFuture('prd - Airbyte - Task-26'), PrefectFuture('prd - Airbyte - Task-27'), PrefectFuture('prd - Airbyte - Task-28'), PrefectFuture('prd - Airbyte - Task-29'), PrefectFuture('prd - Airbyte - Task-30'), PrefectFuture('prd - Airbyte - Task-31'), PrefectFuture('prd - Airbyte - Task-32'), PrefectFuture('prd - Airbyte - Task-33'), PrefectFuture('prd - Airbyte - Task-34'), PrefectFuture('prd - Airbyte - Task-35'), PrefectFuture('prd - Airbyte - Task-36'), PrefectFuture('prd - Airbyte - Task-37'), PrefectFuture('prd - Airbyte - Task-38'), PrefectFuture('prd - Airbyte - Task-39'), PrefectFuture('prd - Airbyte - Task-40'), PrefectFuture('prd - Airbyte - Task-41'), PrefectFuture('prd - Airbyte - Task-42'), PrefectFuture('prd - Airbyte - Task-43'), PrefectFuture('prd - Airbyte - Task-44'), PrefectFuture('prd - Airbyte - Task-45'), PrefectFuture('prd - Airbyte - Task-46'), PrefectFuture('prd - Airbyte - Task-47'), PrefectFuture('prd - Airbyte - Task-48'), PrefectFuture('prd - Airbyte - Task-49'), PrefectFuture('prd - Airbyte - Task-50'), PrefectFuture('prd - Airbyte - Task-51'), PrefectFuture('prd - Airbyte - Task-52'), PrefectFuture('prd - Airbyte - Task-53'), PrefectFuture('prd - Airbyte - Task-54'), PrefectFuture('prd - Airbyte - Task-55'), PrefectFuture('prd - Airbyte - Task-56'), PrefectFuture('prd - Airbyte - Task-57'), PrefectFuture('prd - Airbyte - Task-58'), PrefectFuture('prd - Airbyte - Task-59'), PrefectFuture('prd - Airbyte - Task-60'), PrefectFuture('prd - Airbyte - Task-61'), PrefectFuture('prd - Airbyte - Task-62'), PrefectFuture('prd - Airbyte - Task-63'), PrefectFuture('prd - Airbyte - Task-64'), PrefectFuture('prd - Airbyte - Task-65'), PrefectFuture('prd - Airbyte - Task-66'), PrefectFuture('prd - Airbyte - Task-67'), PrefectFuture('prd - Airbyte - Task-68'), PrefectFuture('prd - Airbyte - Task-69'), PrefectFuture('prd - Airbyte - Task-70'), PrefectFuture('prd - Airbyte - Task-71'), PrefectFuture('prd - Airbyte - Task-72'), PrefectFuture('prd - Airbyte - Task-73'), PrefectFuture('prd - Airbyte - Task-74'), PrefectFuture('prd - Airbyte - Task-75'), PrefectFuture('prd - Airbyte - Task-76'), PrefectFuture('prd - Airbyte - Task-77'), PrefectFuture('prd - Airbyte - Task-78'), PrefectFuture('prd - Airbyte - Task-79'), PrefectFuture('prd - Airbyte - Task-80'), PrefectFuture('prd - Airbyte - Task-81'), PrefectFuture('prd - Airbyte - Task-82'), PrefectFuture('prd - Airbyte - Task-83'), PrefectFuture('prd - Airbyte - Task-84'), PrefectFuture('prd - Airbyte - Task-85'), PrefectFuture('prd - Airbyte - Task-86'), PrefectFuture('prd - Airbyte - Task-87'), PrefectFuture('prd - Airbyte - Task-88'), PrefectFuture('prd - Airbyte - Task-89'), PrefectFuture('prd - Airbyte - Task-90'), PrefectFuture('prd - Airbyte - Task-91'), PrefectFuture('prd - Airbyte - Task-92'), PrefectFuture('prd - Airbyte - Task-93'), PrefectFuture('prd - Airbyte - Task-94'), PrefectFuture('prd - Airbyte - Task-95'), PrefectFuture('prd - Airbyte - Task-96'), PrefectFuture('prd - Airbyte - Task-97'), PrefectFuture('prd - Airbyte - Task-98'), PrefectFuture('prd - Airbyte - Task-99'), PrefectFuture('prd - Airbyte - Task-100'), PrefectFuture('prd - Airbyte - Task-101'), PrefectFuture('prd - Airbyte - Task-102'), PrefectFuture('prd - Airbyte - Task-103'), PrefectFuture('prd - Airbyte - Task-104'), PrefectFuture('prd - Airbyte - Task-105'), PrefectFuture('prd - Airbyte - Task-106'), PrefectFuture('prd - Airbyte - Task-107'), PrefectFuture('prd - Airbyte - Task-108'), PrefectFuture('prd - Airbyte - Task-109'), PrefectFuture('prd - Airbyte - Task-110'), PrefectFuture('prd - Airbyte - Task-111'), PrefectFuture('prd - Airbyte - Task-112'), PrefectFuture('prd - Airbyte - Task-113'), PrefectFuture('prd - Airbyte - Task-114'), PrefectFuture('prd - Airbyte - Task-115'), PrefectFuture('prd - Airbyte - Task-116'), PrefectFuture('prd - Airbyte - Task-117'), PrefectFuture('prd - Airbyte - Task-118'), PrefectFuture('prd - Airbyte - Task-119'), PrefectFuture('prd - Airbyte - Task-120'), PrefectFuture('prd - Airbyte - Task-121'), PrefectFuture('prd - Airbyte - Task-122'), PrefectFuture('prd - Airbyte - Task-123'), PrefectFuture('prd - Airbyte - Task-124'), PrefectFuture('prd - Airbyte - Task-125'), PrefectFuture('prd - Airbyte - Task-126'), PrefectFuture('prd - Airbyte - Task-127'), PrefectFuture('prd - Airbyte - Task-128'), PrefectFuture('prd - Airbyte - Task-129'), PrefectFuture('prd - Airbyte - Task-130'), PrefectFuture('prd - Airbyte - Task-131'), PrefectFuture('prd - Airbyte - Task-132'), PrefectFuture('prd - Airbyte - Task-133'), PrefectFuture('prd - Airbyte - Task-134'), PrefectFuture('prd - Airbyte - Task-135'), PrefectFuture('prd - Airbyte - Task-136'), PrefectFuture('prd - Airbyte - Task-137'), PrefectFuture('prd - Airbyte - Task-138'), PrefectFuture('prd - Airbyte - Task-139'), PrefectFuture('prd - Airbyte - Task-140'), PrefectFuture('prd - Airbyte - Task-141'), PrefectFuture('prd - Airbyte - Task-142'), PrefectFuture('prd - Airbyte - Task-143'), PrefectFuture('prd - Airbyte - Task-144'), PrefectFuture('prd - Airbyte - Task-145'), PrefectFuture('prd - Airbyte - Task-146'), PrefectFuture('prd - Airbyte - Task-147'), PrefectFuture('prd - Airbyte - Task-148'), PrefectFuture('prd - Airbyte - Task-149'), PrefectFuture('prd - Airbyte - Task-150'), PrefectFuture('prd - Airbyte - Task-151'), PrefectFuture('prd - Airbyte - Task-152'), PrefectFuture('prd - Airbyte - Task-153'), PrefectFuture('prd - Airbyte - Task-154'), PrefectFuture('prd - Airbyte - Task-155'), PrefectFuture('prd - Airbyte - Task-156'), PrefectFuture('prd - Airbyte - Task-157'), PrefectFuture('prd - Airbyte - Task-158'), PrefectFuture('prd - Airbyte - Task-159'), PrefectFuture('prd - Airbyte - Task-160'), PrefectFuture('prd - Airbyte - Task-161'), PrefectFuture('prd - Airbyte - Task-162'), PrefectFuture('prd - Airbyte - Task-163'), PrefectFuture('prd - Airbyte - Task-164'), PrefectFuture('prd - Airbyte - Task-165'), PrefectFuture('prd - Airbyte - Task-166'), PrefectFuture('prd - Airbyte - Task-167'), PrefectFuture('prd - Airbyte - Task-168'), PrefectFuture('prd - Airbyte - Task-169'), PrefectFuture('prd - Airbyte - Task-170'), PrefectFuture('prd - Airbyte - Task-171'), PrefectFuture('prd - Airbyte - Task-172'), PrefectFuture('prd - Airbyte - Task-173'), PrefectFuture('prd - Airbyte - Task-174'), PrefectFuture('prd - Airbyte - Task-175'), PrefectFuture('prd - Airbyte - Task-176'), PrefectFuture('prd - Airbyte - Task-177'), PrefectFuture('prd - Airbyte - Task-178'), PrefectFuture('prd - Airbyte - Task-179'), PrefectFuture('prd - Airbyte - Task-180'), PrefectFuture('prd - Airbyte - Task-181'), PrefectFuture('prd - Airbyte - Task-182'), PrefectFuture('prd - Airbyte - Task-183'), PrefectFuture('prd - Airbyte - Task-184'), PrefectFuture('prd - Airbyte - Task-185'), PrefectFuture('prd - Airbyte - Task-186'), PrefectFuture('prd - Airbyte - Task-187'), PrefectFuture('prd - Airbyte - Task-188'), PrefectFuture('prd - Airbyte - Task-189'), PrefectFuture('prd - Airbyte - Task-190'), PrefectFuture('prd - Airbyte - Task-191')]
10:14:41 AM
prefect.flow_runs
Created task run 'prd - Airbyte - Task-0' for task 'prd - Airbyte - Task'
10:14:41 AM
prefect.flow_runs
Submitted task run 'prd - Airbyte - Task-0' for execution.
10:14:42 AM
prefect.flow_runs
Created task run 'prd - Airbyte - Task-5' for task 'prd - Airbyte - Task'
10:14:42 AM
prefect.flow_runs
Submitted task run 'prd - Airbyte - Task-5' for execution.
10:14:43 AM
prefect.flow_runs
Created task run 'prd - Airbyte - Task-3' for task 'prd - Airbyte - Task'
10:14:43 AM
prefect.flow_runsAt this point, it appears that we need to switch to a net-new workspace - our old workspace appears to be corrupted
👀 1
j
Jake Kaplan
08/15/2023, 3:07 PMhey, do you have any task run concurrency limits defined?
z
Zachary Loertscher
08/15/2023, 3:09 PMNo, not that I'm aware of. Would this be the right place to see that?
Wait, I think I found it...
j
Jake Kaplan
08/15/2023, 3:10 PMScreenshot 2023-08-15 at 11.09.58 AM.png
z
Zachary Loertscher
08/15/2023, 3:10 PMAhhhh wow - looks like we did have that in place
So sorry - thank you so much for pointing that out
is there no way to get rid of all the ghost tasks that are still "cancelling"?
I deleted a lot of flow-runs yesterday that were stuck in "cancelling"
j
Jake Kaplan
08/15/2023, 3:13 PMYou can delete any flow runs you'd like!
Before you kick off some new runs you'll probably need to reset the concurrency slots, so they're not blocking your new tasks
Hard deleting a flow doesn't give an opportunity for the task run concurrency to be cleared
✅ 1
🙌 1
z
Zachary Loertscher
08/15/2023, 3:14 PMGotcha - that's really good to know. Thanks a ton @Jake Kaplan! Saved me from a bunch of unnecessary work!
@Jake Kaplan looks like my work-queue is unhealthy now... do task concurrency limits affect the agent health?
n
Nate
08/15/2023, 3:27 PMyes @Zachary Loertscher - if you have tasks that pile up because all concurrency slots are filled, then you'll get an unhealthy agent at this time
d
Deceivious
08/15/2023, 3:41 PM🥺
In an alternate world
Where task concurrency tags simply works
And I don't have to monitor them,
I am happy.
n
Nate
08/15/2023, 3:42 PMhey @Deceivious - your alternate world is not so far away
d
Deceivious
08/15/2023, 3:43 PMBut this is new concurrency and I have to rewrite flows 😔 m lazy 😂
z
Zachary Loertscher
08/16/2023, 5:42 PMHey all - I know this is a bit old, but is there any way for me to re-enable the task-concurrency?
I deleted a bunch of flow runs stuck in cancelling, and I'm assuming those are still "running" & taking up my task concurrency. Can I get rid of those, somehow, so that I can take advantage of task-concurrency in the future? Or will I just not be able to use it going forward?
j
Jake Kaplan
08/16/2023, 6:02 PMHey you should be able to clear the concurrency limit by deleting it and recreating it. Or using
$ prefect concurrency-limit reset <tag>z
Zachary Loertscher
08/16/2023, 6:02 PMAh, so the task-concurrency needed re-setting - sounds like the deleted flow-runs won't be considered after resetting it?
j
Jake Kaplan
08/16/2023, 6:04 PMright, a task run concurrency slot is grabbed when a task run starts and released when it reaches a terminal state.
A common issue with v1 of concurrency limits is what you're seeing, something basically pulls the plug, and the task runs don't have a chance to release their slots (think deleting the flow run or an OOM error)
z
Zachary Loertscher
08/16/2023, 6:05 PMAh I see - that totally makes sense. Thanks for shedding light on that, glad to hear that I can still use task-concurrency limits!
🙌 1