https://prefect.io logo
d

Deceivious

08/04/2023, 9:29 AM
Hi Prefect Devs, Is there a public roadmap / list of bugs and features currently being worked on?
c

Chris Reuter

08/04/2023, 12:01 PM
hi @Deceivious - I responded on something similar the other day. We do track/respond to issues in GitHub, but roadmap level items we don't have a public place for.
🫡 1
d

Deceivious

08/04/2023, 12:10 PM
Thanks, @Chris Reuter . Totally understandable. I was wondering about bugs related to Concurrency Task Limits. 6 out of 26 bugs labelled high priority in github are concurrency task limits related; Oldest one being over a Year old now 😅 Having to manually clear limits and monitor them is starting to be a pain for me. Would you have any ideas about the timeline on this?
c

Chris Reuter

08/04/2023, 12:13 PM
Thank you, I will see if the product team or any OSS specific engineers can weigh in here!
d

Deceivious

08/04/2023, 12:14 PM
Will just add this here. Major issues I am facing: [All of which have their tickets in github] On server restart / worker force restart, tasks that have the lock are not released. Random tasks not releasing the locks - at some point all the limits are taken up - flows are perpetually waiting for the lock until queue health goes bad [if queue is concurrency limited] OR too many pods being spawned in worker server. 👀
thanks @Chris Reuter
j

Jenny

08/04/2023, 1:27 PM
Hey @Deceivious - a quick update that we're actively working on a new improved version of concurrency. Look out for more updates in the #announcements channel in the next few weeks.
🙌 1
d

Deceivious

08/04/2023, 1:28 PM
THanks for the update.
Can I expect it to only affect the back end and not implementation of the flow codes?
j

Jenny

08/04/2023, 2:10 PM
Great question. Let me check with the team who are working on it.
c

Chris Pickett

08/04/2023, 2:23 PM
Hi @Deceivious! I’m one of the devs working on the new concurrency implementation. The issue you call out (that locks aren’t always released) is one of the things we’re attempting to directly address. There’s a couple changes that I believe will help alleviate the pain: 1. The new concurrency feature is a general purpose feature, it has a
concurrency
context manager that’s not tied directly to a flow / task. 2. Concurrency limits can have ‘slot decay’, so if you have a flow run that fails often or you need rate-limiting like capability, you can have the locks on the limit decay over time, rather than be explicitly released. However we will still have issues where if code completely crashes then locks will not be released. To that end the new
concurrency
context manager emits events when it acquires and releases locks and will have related objects like the flow, task, deployment, etc. which should at least make it easier to find which ones have failed. The UI isn’t fully designed yet, but we also want to include a button so you can manually unlock slots or even disable a concurrency limit so it will no longer affect runs. To that end it will affect implementations as it’s a completely separate feature from the current implementation. Let me know if you have any more questions!
🙏 1
d

Deceivious

08/04/2023, 2:28 PM
I guess I have seen the new concurrency code usage in one of the github issues comments. So This would require a change in the flow codes. Would it not be possible to override the working of the task concurrency limits with the new
prefect.concurrency
module internally in prefect so the user code will seamlessly work?
c

Chris Pickett

08/04/2023, 2:39 PM
It would be possible and something we’ve discussed. For the time being, we’ve decided to release it as its own feature, in part because it’s new semi-experimental code, but also it’s fundamentally different than the existing feature. It’s possible we’ll decide to replace the current implementation entirely, but that’s not currently on the roadmap.
d

Deceivious

08/08/2023, 7:46 AM
hi @Chris Pickett I see that the concurrency v2 has been released now. Related github issues outlines how to use concurrency v2 on the flows but not how the concurrency v2 limits can be set up onto the server/ the database. Can you provide some codes to set it up? Also I do not new API route collections in the
/api/docs
. How does a user read / edit concurrency v2 limits via the API?
c

Chris Pickett

08/10/2023, 1:44 AM
Hi @Deceivious it’s not completely launched yet, it’l likely to be another couple weeks, we need to build out a UI for it and work out some sync/async kinks. The big pieces are there, but it’s subject to change for the time being.
🙌 1
Hey @Deceivious it took longer than I expected, but we just released global concurrency limits, see the docs over here: https://docs.prefect.io/2.13.0/guides/global-concurrency-limits/
👀 1
d

Deceivious

09/08/2023, 1:26 PM
Better late than never. 😄 looking into the docs now
I will def. have more question son this . Will get back to u in a while
👍 1
c

Chris Pickett

09/08/2023, 1:26 PM
Awesome, let me know if you run into issues or have questions.
d

Deceivious

09/08/2023, 1:27 PM
Uhh, there is no version tag in the docs. Which version has the feature added in?
c

Chris Pickett

09/08/2023, 1:27 PM
Latest, 2.13.0
d

Deceivious

09/08/2023, 1:33 PM
The doc is still missing how to create server side concurrency object.
c

Chris Pickett

09/08/2023, 1:34 PM
It’s mentioned in the
managing
section: https://docs.prefect.io/2.13.0/guides/global-concurrency-limits/#managing-global-concurrency-limits-and-rate-limits they’re created / edited, etc. from the UI.
In the new UI it’s in the workspace settings under
concurrency
.
d

Deceivious

09/08/2023, 1:39 PM
How about using python api client?
image.png
Also what is the resolution of the slot decay?
c

Chris Pickett

09/08/2023, 1:56 PM
The client doesn’t currently have that method on it, but we’ll likely add a CLI component and part of that would be creating those methods.
d

Deceivious

09/08/2023, 1:56 PM
my code is 99.99% python and 0.01% yaml xD
c

Chris Pickett

09/08/2023, 1:56 PM
Can you say more about the resolution of slot decay?
d

Deceivious

09/08/2023, 1:56 PM
no cli
The slot decay is a float but whats the resolution of the float?
0.000000000000000000000000000000000000001
c

Chris Pickett

09/08/2023, 1:57 PM
Cool, when we add cli support we’ll also need to add client methods that you could use in your python.
d

Deceivious

09/08/2023, 1:58 PM
This is interesting 😄
😂 1
c

Chris Pickett

09/08/2023, 1:58 PM
You certainly could set it to that, but that would be the number of slots that decay per second so you might end up waiting a million years or something to have an open slot.
😂 1
d

Deceivious

09/08/2023, 1:59 PM
Prefect task : Depower my cryogenic pod after 2k years ;
c

Chris Pickett

09/08/2023, 1:59 PM
Hahah, I’d use a scheduled deployment for that 😉
d

Deceivious

09/08/2023, 2:01 PM
OK From what I understand the new concurrency tag v2 will be deallocated from a task when: The code flow exits the context OR The time allocated using the "slot decay" runs out Am i correct?
c

Chris Pickett

09/08/2023, 2:03 PM
Right, if you have slot decay configured (which isn’t required). The
concurrency
context manager will acquire a slot, and block code execution until it’s able to get one, then it’ll let your code run, and when your code exits the context manager it’ll release that slot back to be used by another process/task/etc. If you have slot decay configured a slot could open up before the concurrency manager explicitly releases it.
d

Deceivious

09/08/2023, 2:04 PM
So when I have the slot decay configured, and the code inside the context is not complete - does it also kill off the task and raise an exception? Or will the code continue to run in the background after losing the slot
c

Chris Pickett

09/08/2023, 2:05 PM
It’ll continue to run.
🙌 1
d

Deceivious

09/08/2023, 2:09 PM
Cool. SO what about the rate limit. Is there some way for me to say run this task only 60 times per min and wait if it exceeds the limit?
The document states rate limit but is that concurrency v2?
or some other server side control mechanism?
c

Chris Pickett

09/08/2023, 2:12 PM
The
rate_limit
function uses the same concurrency limits as the
concurrency
context manager. It uses that same slot decay mechanism, so instead of explicitly releasing a slot it just uses the decay for management of the active slots. You can get something to run just 60 times per min, in my testing the best way I’ve found to make something like that happen is to create a Concurrency limit with a limit of
1
and a slot decay of
1
, that would mean that the slot will be available again every second, but feel free to mess around with a combination of slot decay and limit to see what works best for your use case.
d

Deceivious

09/08/2023, 2:12 PM
Got it!
An actual rate limit would have been great tho
c

Chris Pickett

09/08/2023, 2:14 PM
Admittedly I haven’t done any real world usage of the rate limiter, but in my testing it’s actually a really powerful concept. It’s not just a rate limit in-process, but a distributed rate limit. So you could spawn 100 tasks and all have them abide the same rate limit.
d

Deceivious

09/08/2023, 2:25 PM
I can understand where you are coming from. but hear me out 😄
If it was a purely distributed rate limiter, I wouldnt have to think about the
limit
. No matter how many tasks i run in parallel , the server would limit all the pending task until a slot is open right?
In this version , I have to think about the limit as well
c

Chris Pickett

09/08/2023, 2:27 PM
Certainly it’s a different way of thinking about it, you can’t just say ‘allow 30 per minute’, you have to consider how to get there with a combination of limits and slot decay.
d

Deceivious

09/08/2023, 2:27 PM
Yes exactly what i mean
We have dynamic number of workflow and the agent come in and out depending on the load of the system
we use k8s
And m thinking about how to set the slot decay in prefect concurrency v2 and the calculation keeps getting more and more complicated 😄
but hey Ill take a concurrency limit that doesnt freeze up forever until manually cleared like the v1 did. Thats a big improvement 😄
🙌 1
@Chris Pickett One last question, Which component of the server is responsible for ensuring that the concurrency slot are cleared in time? I dont see any new table on slot level in the database. ie there is not table to say which task acquired a slot at what time. Howw is this being managed?
c

Chris Pickett

09/08/2023, 2:35 PM
It doesn’t track which task acquired which slot, the database itself only knows how many slots it has as a limit and how many are ‘active’.
d

Deceivious

09/08/2023, 2:35 PM
Ah got it.
c

Chris Pickett

09/08/2023, 2:36 PM
Events are emitted when a slot is acquired and if it’s acquired while a flow/task is running that flow/task will be a related object on that event.
But one of the goals with this project was to make global concurrency useful outside of a flow.
d

Deceivious

09/08/2023, 2:39 PM
Arent Prefect Event a cloud only feature?
c

Chris Pickett

09/08/2023, 2:39 PM
Yes they are
d

Deceivious

09/08/2023, 2:39 PM
image.png