https://prefect.io logo
Join Slack
Powered by
Hi all - I have a prefect job that kicks off airby...
# ask-community
z
Hi all - I have a prefect job that kicks off airbyte connections. However, after creating all the tasks, they all hang in "pending" These airbyte connections sit on an EC2 - I've rebooted it a couple of times to no avail. Prefect agent is healthy. Including logs in thread - any ideas? Is there something I need to re-set on the prefect side? I had some flows yesterday that got stuck in "cancelling" that I deleted - could that be causing my tasks to get stuck in pending? Please help! Prefect version - 2.8.7
✅ 1
Copy code
Downloading flow code from storage at ''
10:14:39 AM
prefect.flow_runs
Created task run '{dev_stage} - fetch airbyte ConnectionIds-0' for task '{dev_stage} - fetch airbyte ConnectionIds'
10:14:40 AM
prefect.flow_runs
Executing '{dev_stage} - fetch airbyte ConnectionIds-0' immediately...
10:14:40 AM
prefect.flow_runs
fetch airbyte connection id's for the prd environment that run on a daily basis
10:14:40 AM
{dev_stage} - fetch airbyte ConnectionIds-0
prefect.task_runs
Finished in state Completed()
10:14:40 AM
{dev_stage} - fetch airbyte ConnectionIds-0
prefect.task_runs
Created task run 'create airbyte connections list-0' for task 'create airbyte connections list'
10:14:40 AM
prefect.flow_runs
Executing 'create airbyte connections list-0' immediately...
10:14:40 AM
prefect.flow_runs
converting airbyte connectionids into a dictionary
10:14:41 AM
create airbyte connections list-0
prefect.task_runs
Finished in state Completed()
10:14:41 AM
create airbyte connections list-0
prefect.task_runs
[PrefectFuture('prd - Airbyte - Task-0'), PrefectFuture('prd - Airbyte - Task-1'), PrefectFuture('prd - Airbyte - Task-2'), PrefectFuture('prd - Airbyte - Task-3'), PrefectFuture('prd - Airbyte - Task-4'), PrefectFuture('prd - Airbyte - Task-5'), PrefectFuture('prd - Airbyte - Task-6'), PrefectFuture('prd - Airbyte - Task-7'), PrefectFuture('prd - Airbyte - Task-8'), PrefectFuture('prd - Airbyte - Task-9'), PrefectFuture('prd - Airbyte - Task-10'), PrefectFuture('prd - Airbyte - Task-11'), PrefectFuture('prd - Airbyte - Task-12'), PrefectFuture('prd - Airbyte - Task-13'), PrefectFuture('prd - Airbyte - Task-14'), PrefectFuture('prd - Airbyte - Task-15'), PrefectFuture('prd - Airbyte - Task-16'), PrefectFuture('prd - Airbyte - Task-17'), PrefectFuture('prd - Airbyte - Task-18'), PrefectFuture('prd - Airbyte - Task-19'), PrefectFuture('prd - Airbyte - Task-20'), PrefectFuture('prd - Airbyte - Task-21'), PrefectFuture('prd - Airbyte - Task-22'), PrefectFuture('prd - Airbyte - Task-23'), PrefectFuture('prd - Airbyte - Task-24'), PrefectFuture('prd - Airbyte - Task-25'), PrefectFuture('prd - Airbyte - Task-26'), PrefectFuture('prd - Airbyte - Task-27'), PrefectFuture('prd - Airbyte - Task-28'), PrefectFuture('prd - Airbyte - Task-29'), PrefectFuture('prd - Airbyte - Task-30'), PrefectFuture('prd - Airbyte - Task-31'), PrefectFuture('prd - Airbyte - Task-32'), PrefectFuture('prd - Airbyte - Task-33'), PrefectFuture('prd - Airbyte - Task-34'), PrefectFuture('prd - Airbyte - Task-35'), PrefectFuture('prd - Airbyte - Task-36'), PrefectFuture('prd - Airbyte - Task-37'), PrefectFuture('prd - Airbyte - Task-38'), PrefectFuture('prd - Airbyte - Task-39'), PrefectFuture('prd - Airbyte - Task-40'), PrefectFuture('prd - Airbyte - Task-41'), PrefectFuture('prd - Airbyte - Task-42'), PrefectFuture('prd - Airbyte - Task-43'), PrefectFuture('prd - Airbyte - Task-44'), PrefectFuture('prd - Airbyte - Task-45'), PrefectFuture('prd - Airbyte - Task-46'), PrefectFuture('prd - Airbyte - Task-47'), PrefectFuture('prd - Airbyte - Task-48'), PrefectFuture('prd - Airbyte - Task-49'), PrefectFuture('prd - Airbyte - Task-50'), PrefectFuture('prd - Airbyte - Task-51'), PrefectFuture('prd - Airbyte - Task-52'), PrefectFuture('prd - Airbyte - Task-53'), PrefectFuture('prd - Airbyte - Task-54'), PrefectFuture('prd - Airbyte - Task-55'), PrefectFuture('prd - Airbyte - Task-56'), PrefectFuture('prd - Airbyte - Task-57'), PrefectFuture('prd - Airbyte - Task-58'), PrefectFuture('prd - Airbyte - Task-59'), PrefectFuture('prd - Airbyte - Task-60'), PrefectFuture('prd - Airbyte - Task-61'), PrefectFuture('prd - Airbyte - Task-62'), PrefectFuture('prd - Airbyte - Task-63'), PrefectFuture('prd - Airbyte - Task-64'), PrefectFuture('prd - Airbyte - Task-65'), PrefectFuture('prd - Airbyte - Task-66'), PrefectFuture('prd - Airbyte - Task-67'), PrefectFuture('prd - Airbyte - Task-68'), PrefectFuture('prd - Airbyte - Task-69'), PrefectFuture('prd - Airbyte - Task-70'), PrefectFuture('prd - Airbyte - Task-71'), PrefectFuture('prd - Airbyte - Task-72'), PrefectFuture('prd - Airbyte - Task-73'), PrefectFuture('prd - Airbyte - Task-74'), PrefectFuture('prd - Airbyte - Task-75'), PrefectFuture('prd - Airbyte - Task-76'), PrefectFuture('prd - Airbyte - Task-77'), PrefectFuture('prd - Airbyte - Task-78'), PrefectFuture('prd - Airbyte - Task-79'), PrefectFuture('prd - Airbyte - Task-80'), PrefectFuture('prd - Airbyte - Task-81'), PrefectFuture('prd - Airbyte - Task-82'), PrefectFuture('prd - Airbyte - Task-83'), PrefectFuture('prd - Airbyte - Task-84'), PrefectFuture('prd - Airbyte - Task-85'), PrefectFuture('prd - Airbyte - Task-86'), PrefectFuture('prd - Airbyte - Task-87'), PrefectFuture('prd - Airbyte - Task-88'), PrefectFuture('prd - Airbyte - Task-89'), PrefectFuture('prd - Airbyte - Task-90'), PrefectFuture('prd - Airbyte - Task-91'), PrefectFuture('prd - Airbyte - Task-92'), PrefectFuture('prd - Airbyte - Task-93'), PrefectFuture('prd - Airbyte - Task-94'), PrefectFuture('prd - Airbyte - Task-95'), PrefectFuture('prd - Airbyte - Task-96'), PrefectFuture('prd - Airbyte - Task-97'), PrefectFuture('prd - Airbyte - Task-98'), PrefectFuture('prd - Airbyte - Task-99'), PrefectFuture('prd - Airbyte - Task-100'), PrefectFuture('prd - Airbyte - Task-101'), PrefectFuture('prd - Airbyte - Task-102'), PrefectFuture('prd - Airbyte - Task-103'), PrefectFuture('prd - Airbyte - Task-104'), PrefectFuture('prd - Airbyte - Task-105'), PrefectFuture('prd - Airbyte - Task-106'), PrefectFuture('prd - Airbyte - Task-107'), PrefectFuture('prd - Airbyte - Task-108'), PrefectFuture('prd - Airbyte - Task-109'), PrefectFuture('prd - Airbyte - Task-110'), PrefectFuture('prd - Airbyte - Task-111'), PrefectFuture('prd - Airbyte - Task-112'), PrefectFuture('prd - Airbyte - Task-113'), PrefectFuture('prd - Airbyte - Task-114'), PrefectFuture('prd - Airbyte - Task-115'), PrefectFuture('prd - Airbyte - Task-116'), PrefectFuture('prd - Airbyte - Task-117'), PrefectFuture('prd - Airbyte - Task-118'), PrefectFuture('prd - Airbyte - Task-119'), PrefectFuture('prd - Airbyte - Task-120'), PrefectFuture('prd - Airbyte - Task-121'), PrefectFuture('prd - Airbyte - Task-122'), PrefectFuture('prd - Airbyte - Task-123'), PrefectFuture('prd - Airbyte - Task-124'), PrefectFuture('prd - Airbyte - Task-125'), PrefectFuture('prd - Airbyte - Task-126'), PrefectFuture('prd - Airbyte - Task-127'), PrefectFuture('prd - Airbyte - Task-128'), PrefectFuture('prd - Airbyte - Task-129'), PrefectFuture('prd - Airbyte - Task-130'), PrefectFuture('prd - Airbyte - Task-131'), PrefectFuture('prd - Airbyte - Task-132'), PrefectFuture('prd - Airbyte - Task-133'), PrefectFuture('prd - Airbyte - Task-134'), PrefectFuture('prd - Airbyte - Task-135'), PrefectFuture('prd - Airbyte - Task-136'), PrefectFuture('prd - Airbyte - Task-137'), PrefectFuture('prd - Airbyte - Task-138'), PrefectFuture('prd - Airbyte - Task-139'), PrefectFuture('prd - Airbyte - Task-140'), PrefectFuture('prd - Airbyte - Task-141'), PrefectFuture('prd - Airbyte - Task-142'), PrefectFuture('prd - Airbyte - Task-143'), PrefectFuture('prd - Airbyte - Task-144'), PrefectFuture('prd - Airbyte - Task-145'), PrefectFuture('prd - Airbyte - Task-146'), PrefectFuture('prd - Airbyte - Task-147'), PrefectFuture('prd - Airbyte - Task-148'), PrefectFuture('prd - Airbyte - Task-149'), PrefectFuture('prd - Airbyte - Task-150'), PrefectFuture('prd - Airbyte - Task-151'), PrefectFuture('prd - Airbyte - Task-152'), PrefectFuture('prd - Airbyte - Task-153'), PrefectFuture('prd - Airbyte - Task-154'), PrefectFuture('prd - Airbyte - Task-155'), PrefectFuture('prd - Airbyte - Task-156'), PrefectFuture('prd - Airbyte - Task-157'), PrefectFuture('prd - Airbyte - Task-158'), PrefectFuture('prd - Airbyte - Task-159'), PrefectFuture('prd - Airbyte - Task-160'), PrefectFuture('prd - Airbyte - Task-161'), PrefectFuture('prd - Airbyte - Task-162'), PrefectFuture('prd - Airbyte - Task-163'), PrefectFuture('prd - Airbyte - Task-164'), PrefectFuture('prd - Airbyte - Task-165'), PrefectFuture('prd - Airbyte - Task-166'), PrefectFuture('prd - Airbyte - Task-167'), PrefectFuture('prd - Airbyte - Task-168'), PrefectFuture('prd - Airbyte - Task-169'), PrefectFuture('prd - Airbyte - Task-170'), PrefectFuture('prd - Airbyte - Task-171'), PrefectFuture('prd - Airbyte - Task-172'), PrefectFuture('prd - Airbyte - Task-173'), PrefectFuture('prd - Airbyte - Task-174'), PrefectFuture('prd - Airbyte - Task-175'), PrefectFuture('prd - Airbyte - Task-176'), PrefectFuture('prd - Airbyte - Task-177'), PrefectFuture('prd - Airbyte - Task-178'), PrefectFuture('prd - Airbyte - Task-179'), PrefectFuture('prd - Airbyte - Task-180'), PrefectFuture('prd - Airbyte - Task-181'), PrefectFuture('prd - Airbyte - Task-182'), PrefectFuture('prd - Airbyte - Task-183'), PrefectFuture('prd - Airbyte - Task-184'), PrefectFuture('prd - Airbyte - Task-185'), PrefectFuture('prd - Airbyte - Task-186'), PrefectFuture('prd - Airbyte - Task-187'), PrefectFuture('prd - Airbyte - Task-188'), PrefectFuture('prd - Airbyte - Task-189'), PrefectFuture('prd - Airbyte - Task-190'), PrefectFuture('prd - Airbyte - Task-191')]
10:14:41 AM
prefect.flow_runs
Created task run 'prd - Airbyte - Task-0' for task 'prd - Airbyte - Task'
10:14:41 AM
prefect.flow_runs
Submitted task run 'prd - Airbyte - Task-0' for execution.
10:14:42 AM
prefect.flow_runs
Created task run 'prd - Airbyte - Task-5' for task 'prd - Airbyte - Task'
10:14:42 AM
prefect.flow_runs
Submitted task run 'prd - Airbyte - Task-5' for execution.
10:14:43 AM
prefect.flow_runs
Created task run 'prd - Airbyte - Task-3' for task 'prd - Airbyte - Task'
10:14:43 AM
prefect.flow_runs
At this point, it appears that we need to switch to a net-new workspace - our old workspace appears to be corrupted
👀 1
j
hey, do you have any task run concurrency limits defined?
z
No, not that I'm aware of. Would this be the right place to see that?
Wait, I think I found it...
j
z
Ahhhh wow - looks like we did have that in place
So sorry - thank you so much for pointing that out is there no way to get rid of all the ghost tasks that are still "cancelling"?
I deleted a lot of flow-runs yesterday that were stuck in "cancelling"
j
You can delete any flow runs you'd like! Before you kick off some new runs you'll probably need to reset the concurrency slots, so they're not blocking your new tasks Hard deleting a flow doesn't give an opportunity for the task run concurrency to be cleared
🙌 1
✅ 1
z
Gotcha - that's really good to know. Thanks a ton @Jake Kaplan! Saved me from a bunch of unnecessary work!
@Jake Kaplan looks like my work-queue is unhealthy now... do task concurrency limits affect the agent health?
n
yes @Zachary Loertscher - if you have tasks that pile up because all concurrency slots are filled, then you'll get an unhealthy agent at this time
d
🥺 In an alternate world Where task concurrency tags simply works And I don't have to monitor them, I am happy.
n
hey @Deceivious - your alternate world is not so far away
d
But this is new concurrency and I have to rewrite flows 😔 m lazy 😂
z
Hey all - I know this is a bit old, but is there any way for me to re-enable the task-concurrency? I deleted a bunch of flow runs stuck in cancelling, and I'm assuming those are still "running" & taking up my task concurrency. Can I get rid of those, somehow, so that I can take advantage of task-concurrency in the future? Or will I just not be able to use it going forward?
j
Hey you should be able to clear the concurrency limit by deleting it and recreating it. Or using 
$ prefect concurrency-limit reset <tag>
z
Ah, so the task-concurrency needed re-setting - sounds like the deleted flow-runs won't be considered after resetting it?
j
right, a task run concurrency slot is grabbed when a task run starts and released when it reaches a terminal state. A common issue with v1 of concurrency limits is what you're seeing, something basically pulls the plug, and the task runs don't have a chance to release their slots (think deleting the flow run or an OOM error)
z
Ah I see - that totally makes sense. Thanks for shedding light on that, glad to hear that I can still use task-concurrency limits!
🙌 1
Open in Slack
PreviousNext
Powered by