Trying to get this PR across the line: https://github.com/PrefectHQ/prefect/actions/runs/19767933876/job/56645184779?pr=19559 But I'm seeing CI fail on python3.10, python3.12 (passes for 11, 13, 14). I suspect flakiness could be a problem since there shouldn't really be differences exposed by this change in terms of python version (just adds a setting). Is this a known problem with the Github CI suite for Prefect?

hi @Sebastian S, unfortunately yea there's a dependency we introduced recently that's caused these 2 to be flaky, hope to fix next week

no worries thanks for the heads up!

you don't need to keep submitting, no. i'll give this a review today, thanks!

Thanks!

hi @Sebastian S - can i ask why you're interested in parametrizing the read_batch_size here? we tend to avoid sprawling settings whenever possible, so i'm curious to hear why you think this is a good knob to expose one problem with the PR is that

read_batch_size

is not a valid kwarg for the default memory consumer. we should correctly ignore it, but its another reason to stop and ask now whether we really need to add these settings

Fair question - in a real-world setting I've observed this increases the Redis read performance by an order of magnitude

thanks, that's helpful context! i think its probably good as is

thanks, no worries

yep! we're chatting about this one internally, its a non-trivial change so we'll have to be careful but it makes sense at a glance

For sure. I'm happy to hammer the unit tests a bit harder, I can also share some real-world performance data on what this unlocks

yea any metrics you can put on the PR body or comments on the PR would be super appreciated we have a rather large HA setup based on dev tags, but before we even merge into main we could smoke test by temporarily running our testbed on an image based on your branch

There is one bit of trickiness here, which isn't changed/exposed by the PR, but should be flagged. I did this so that we can vertically scale taskRecorderService, but also I am horizontally scaling it in my setup to churn through the Redis event queue faster. However horizontally scaling it does cause causality to be violated more often (if A depends on B, they might now be read by different consumers). The code is already written to be eventually consistent, so that A will eventually be persisted even if B is never seen, so that's ok, but just flagging it.