Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

This is awesome and should be included with Prefect itself, it fills a pretty big gap where you need to run a bunch of CPU-bound tasks on a VM with lots of CPU and RAM. Ray and Dask are overkill (and fussy).

<@U078RAN2UQ0> Thanks! It sounds like your use case mirrors my own. I wanted to run CPU-bound tasks using Python's multiprocessing builtins without needing to pull in Dask or Ray. Not that they aren't great - just, like you said, they feel like overkill for just running things locally.

There are still some bugs to work out that only appear under certain circumstances. I just released a new beta that fixes one of them! If you have a chance, please feel free to install v0.2.0b4 and see if it handles your workloads. If you run into any problems, it'd be great if you could open a GitHub issue.

hi <@U078RAN2UQ0> <@U03NW4JEYLD>

i left a <https://github.com/PrefectHQ/prefect/issues/12075#issuecomment-3181502449|comment> on this older issue indicating that it might be worth folding this into prefect proper

(thanks again Ryan for taking the initiative on this :muscle: :catjam: and sorry for the lethargy)

<@U02GMEZU18B> There's one hacky part I think I'd need to improve before it's ready for that. There's an open issue for it and I'm working through it now, but I'd be happy to work on a PR once that's resolved if you want to go that route

heck yea. sounds good. feel free to open a draft PR whenever you like, happy to work through it on GH

yea looking at your deps, this would be a no cost improvement to add alongside the existing task runners imo

in full disclosure, we have a couple tweaks to make around api calls in child processes (like broadly, in all task runners that use separate processes) that we might want to take an opportunity to address before merging, but i see no rush here anyways :slightly_smiling_face:

Sounds good - I can make any changes needed improvements pre-merge. This was my first crack at porting the multiprocess task runner from Prefect 2 to 3 and I haven't had much time to revisit it since June. Should have a good block of time starting later this week though. I'm sure there are multiple areas I can improve!

I'll fix what I can and take a look at the latest prefect-dask and prefect-ray changes to see if I can learn from any recent changes there. Then I'll open the PR and we can work through any remaining issues

<https://github.com/PrefectHQ/prefect/pull/18724|heads up!>

alex came through with a process pool impl that should go out later this week

if you have any thoughts on the implementation based on your exploration thus far, feel free to open an issue or comment here!