Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

I have begun consolidating and standardizing Prefect code that I reuse across projects. I decided to package them into my first Python package, <https://github.com/dominictarro/prefecto|prefecto>. Learned a bunch of new tools on the way and would love some feedback!

The batching seems great!! I'm going to experiment with it. The concurrent task runner reliably crashes if fed more than a few hundred tasks at a time, even if execution is limited by a semaphore. Have been meaning to write a script to do this kind of batching

Curious if you've run any tests to see how much slower execution is with the batch implementation

In some informal experiments just now, it seems plenty fast. Good stuff!

Just noticed this, and the batch implementation was just what I was looking for. Thanks for both writing and telling us about this <@U0495JG8RB7>

Came across this after running into <https://github.com/PrefectHQ/prefect/issues/9837|the issue described here>. The batching functionality was exactly what I needed - thanks for this <@U0495JG8RB7>!