Hi there, my name is Derek and I’m an engineer at Snorkel.ai. We build a platform for data centric AI and we run a lot of jobs, ranging from data preprocessing to computing metrics to making queries to external LLM providers. I’m currently exploring how we can redesign our job system away from running monolithic functions, but rather running as a series of blocks (tasks and subflows!). Each block might have different resource constraints — one task might be CPU bound and sent to execute on our Ray cluster, one job might be making network requests and just needs to run multithreaded. I came across Prefect because it can get messy managing the dependencies between all these blocks. Prefect seems like a great way to manage the dependencies while at the same time providing robust features for concurrency + retries. I’m curious if this community has thoughts on if Prefect is the right tool for this type of problem or if it is overkill. It would be running entirely in process (an RQ worker would execute the Flow) and if there are any tradeoffs/considerations that I should be aware of before we start developing

Hey Derek! I love this use case, and in fact it's one I was discussing with some folks earlier today; here's my take: first and foremost, I do think Prefect is the right tool and there are a number of large Prefect deployments optimized for this type of situation. What you're most likely going to find is that Prefect contains all of the composable pieces to achieve this outcome, but you might have to make some subjective choices around how to put them together as I don't think there are obvious best practices documented currently. In fact, we just (quietly) put out a release candidate (

3.0.0rc1

on pypi) that covers more surface area in this realm -- specifically, improved performance and a first class task scheduling service (so you can now orchestrate individual tasks in-process or remotely outside of a flow context). If you're game, I'd love to talk through this with you and see if I can help; given this is something I'm actively thinking about I may be able to turn around any feedback relatively quickly. Let me know if you'd be up for it!

Yes, would love to chat more. Do you have some time next week, Thurs/Fri? The week after works as well

Yup I'll DM you to set something up

I’m interested in using Prefect in a similar way. If at all possible, would love to learn about the take-aways of your discussion that you could share.