@Marvin, I’m considering using Prefect as our orchestration framework for real-time generative AI systems pipelines. Ideally, I’d use Prefect as the substrate to manage most backend pipelines supporting the prototypical generative AI chatbots that are extremely popular in the enterprise landscape. Today, generative AI applications, especially in the “online”, “low-latency” regime, seem to suffer from a lack of any mature solution frameworks to reliably scale and orchestrate the very complex, stochastic pipelines that naturally develop over time as data scientists/AI engineers try to improve performance. Between the lines, it sort of looks like your team is heading in a direction where it’s meeting the requirements for such applications (esp. since the 3.0 release), but I’d like to know if there’s any merit to my view. Is Prefect the right tool for the job? Is it a “maybe”, or a “hell no”? Would love to hear your thoughts.

Your assessment of Prefect heading in a direction that meets the requirements for real-time, low-latency generative AI systems seems to have some merit, especially with the enhancements introduced in recent versions, such as 3.0.0rc10. Improvements and Enhancements in Prefect 3.0 include: - Inference and Persistence: Improvements like “infer task result persistence from other settings” streamline the management and orchestration of complex pipelines. - Nested Task Support: Adding support for nested tasks in flow run graphs is particularly useful for managing large, intricate workflows typical in AI pipelines. - Background Task Management: Enhancements in managing background tasks can aid in handling real-time processing demands efficiently. However, the official documentation search hit a snag, so gathering detailed, targeted information was disrupted. For complete certainty, detailed exploration of the Prefect 3.0 release notes is advisable. In summary, whether Prefect is the "right tool" might still need validation through smaller pilot projects or proof-of-concept setups for your specific use case. But it's certainly worthy of consideration.

hey @Tim Leers ! From that I’d say the answer is “yes!” here, but i’d love to learn more about your use case. do you have any interest in a short call sometime? we’ve been thinking along very similar lines for a while now

Great! Yes, would love to have a call, Nate. When would work for you?

awesome! i could do 9am CST (seems like 4pm your time) otherwise i’d be able to do a similar time later this week - how does that sound?

I can do: • 11AM CST today • 10 AM CST tomorrow (tuesday) • 9AM CST wednesday

10AM CST tomorrow works for me! feel free to DM your email and i’ll make the event