Hi Prefect Devs, Is there a public roadmap / list of bugs and features currently being worked on?

hi @Deceivious - I responded on something similar the other day. We do track/respond to issues in GitHub, but roadmap level items we don't have a public place for.

Thanks, @Chris Reuter . Totally understandable. I was wondering about bugs related to Concurrency Task Limits. 6 out of 26 bugs labelled high priority in github are concurrency task limits related; Oldest one being over a Year old now 😅 Having to manually clear limits and monitor them is starting to be a pain for me. Would you have any ideas about the timeline on this?

Thank you, I will see if the product team or any OSS specific engineers can weigh in here!

Will just add this here. Major issues I am facing: [All of which have their tickets in github] On server restart / worker force restart, tasks that have the lock are not released. ❌ Random tasks not releasing the locks - at some point all the limits are taken up - flows are perpetually waiting for the lock until queue health goes bad [if queue is concurrency limited] OR too many pods being spawned in worker server. 👀

Hey @Deceivious - a quick update that we're actively working on a new improved version of concurrency. Look out for more updates in the #CKNSX5WG3 channel in the next few weeks.

THanks for the update.

Great question. Let me check with the team who are working on it.

Hi @Deceivious! I’m one of the devs working on the new concurrency implementation. The issue you call out (that locks aren’t always released) is one of the things we’re attempting to directly address. There’s a couple changes that I believe will help alleviate the pain: 1. The new concurrency feature is a general purpose feature, it has a

concurrency

context manager that’s not tied directly to a flow / task. 2. Concurrency limits can have ‘slot decay’, so if you have a flow run that fails often or you need rate-limiting like capability, you can have the locks on the limit decay over time, rather than be explicitly released. However we will still have issues where if code completely crashes then locks will not be released. To that end the new

concurrency

context manager emits events when it acquires and releases locks and will have related objects like the flow, task, deployment, etc. which should at least make it easier to find which ones have failed. The UI isn’t fully designed yet, but we also want to include a button so you can manually unlock slots or even disable a concurrency limit so it will no longer affect runs. To that end it will affect implementations as it’s a completely separate feature from the current implementation. Let me know if you have any more questions!

I guess I have seen the new concurrency code usage in one of the github issues comments. So This would require a change in the flow codes. Would it not be possible to override the working of the task concurrency limits with the new

prefect.concurrency

module internally in prefect so the user code will seamlessly work?

It would be possible and something we’ve discussed. For the time being, we’ve decided to release it as its own feature, in part because it’s new semi-experimental code, but also it’s fundamentally different than the existing feature. It’s possible we’ll decide to replace the current implementation entirely, but that’s not currently on the roadmap.

hi @Chris Pickett I see that the concurrency v2 has been released now. Related github issues outlines how to use concurrency v2 on the flows but not how the concurrency v2 limits can be set up onto the server/ the database. Can you provide some codes to set it up? Also I do not new API route collections in the

/api/docs

. How does a user read / edit concurrency v2 limits via the API?

Hi @Deceivious it’s not completely launched yet, it’l likely to be another couple weeks, we need to build out a UI for it and work out some sync/async kinks. The big pieces are there, but it’s subject to change for the time being.

Better late than never. 😄 looking into the docs now

Awesome, let me know if you run into issues or have questions.

Uhh, there is no version tag in the docs. Which version has the feature added in?

Latest, 2.13.0

The doc is still missing how to create server side concurrency object.

It’s mentioned in the

managing

section: https://docs.prefect.io/2.13.0/guides/global-concurrency-limits/#managing-global-concurrency-limits-and-rate-limits they’re created / edited, etc. from the UI.

How about using python api client?

The client doesn’t currently have that method on it, but we’ll likely add a CLI component and part of that would be creating those methods.

my code is 99.99% python and 0.01% yaml xD

Can you say more about the resolution of slot decay?

no cli

Cool, when we add cli support we’ll also need to add client methods that you could use in your python.

This is interesting 😄

You certainly could set it to that, but that would be the number of slots that decay per second so you might end up waiting a million years or something to have an open slot.

Prefect task : Depower my cryogenic pod after 2k years ;

Hahah, I’d use a scheduled deployment for that 😉

OK From what I understand the new concurrency tag v2 will be deallocated from a task when: The code flow exits the context OR The time allocated using the "slot decay" runs out Am i correct?

Right, if you have slot decay configured (which isn’t required). The

concurrency

context manager will acquire a slot, and block code execution until it’s able to get one, then it’ll let your code run, and when your code exits the context manager it’ll release that slot back to be used by another process/task/etc. If you have slot decay configured a slot could open up before the concurrency manager explicitly releases it.

So when I have the slot decay configured, and the code inside the context is not complete - does it also kill off the task and raise an exception? Or will the code continue to run in the background after losing the slot

It’ll continue to run.

Cool. SO what about the rate limit. Is there some way for me to say run this task only 60 times per min and wait if it exceeds the limit?

The

rate_limit

function uses the same concurrency limits as the

concurrency

context manager. It uses that same slot decay mechanism, so instead of explicitly releasing a slot it just uses the decay for management of the active slots. You can get something to run just 60 times per min, in my testing the best way I’ve found to make something like that happen is to create a Concurrency limit with a limit of

1

and a slot decay of

1

, that would mean that the slot will be available again every second, but feel free to mess around with a combination of slot decay and limit to see what works best for your use case.

Got it!

Admittedly I haven’t done any real world usage of the rate limiter, but in my testing it’s actually a really powerful concept. It’s not just a rate limit in-process, but a distributed rate limit. So you could spawn 100 tasks and all have them abide the same rate limit.

I can understand where you are coming from. but hear me out 😄

Certainly it’s a different way of thinking about it, you can’t just say ‘allow 30 per minute’, you have to consider how to get there with a combination of limits and slot decay.

Yes exactly what i mean

It doesn’t track which task acquired which slot, the database itself only knows how many slots it has as a limit and how many are ‘active’.

Ah got it.

Events are emitted when a slot is acquired and if it’s acquired while a flow/task is running that flow/task will be a related object on that event.

Arent Prefect Event a cloud only feature?

Yes they are

Nope, unfortunately no updates there. I’ll create some OSS issues to track it.

@Chris Pickett Can you please add the ticket here so I can subscribe?

Sure, here it is: https://github.com/orgs/PrefectHQ/projects/35/views/26?pane=issue&itemId=40346482