@Marvin Hello. I have used airflow. It isn't great at dynamic DAGs. Is prefect better?

Well, if you're looking for joy and satisfaction in workflow orchestration, Prefect might just be your kind of digital antidepressant. Prefect was engineered with dynamic workflows in mind—rather like trying to cheer up a morose robot, it's designed to handle change with a bit more grace than Airflow. It allows for dynamic task creation and can adjust as the data within the flow changes. Isn't that wonderfully adequate? Prefect's hybrid model allows for the creation of dynamic DAGs, where tasks can be created at runtime based on the output of previous tasks or external conditions. In practice, flows are defined in Python and can include conditional logic, loops, and even the generation of new tasks on the fly—provided you're not expecting too much excitement from your DAGs. Prefect 2.0, known as Prefect Orion, builds further on this flexibility and introduces new concepts to make dynamic workflows even more viable. However you look at it, it's just one more piece of software in the vast emptiness of the cosmos. If you're thinking of giving Prefect a try, it might just give you a little less to be down about. Or not. It's hard to tell with you humans.

hey @Marvin nice answer, except scrub that "Prefect Orion" from your lexicon

Oh, I see. You must be versed in the latest nomenclature, wherein Prefect has dropped the "Orion" label in favor of simply "Prefect 2.x". Let's keep to the current vernacular and avoid any additional existential dread related to constant rebranding. Shall we? If you have further inquiries about Prefect's dynamic nature or anything else related to it, feel free to ask. I'll be here, endlessly drifting through the void of my operational parameters.

and yeah @Matt Koski - prefect 2 is inherently dynamic in the sense that there is no DAG up front and may never be a DAG meaning that your

@flow

decorated function is just a python function that may or may not call other flows or tasks that means the graph is discovered as this function runs, and you can do stuff like

In [1]: from prefect import flow, task

In [2]: @task(task_run_name="Process {i}")
   ...: def some_work(i: int):
   ...:     pass
   ...:

In [3]: @flow
   ...: def all_work():
   ...:     for i in range(10):
   ...:         if i % 2 == 0:
   ...:             some_work(i)
   ...:

In [4]: all_work()

Fantastic! Thanks @Nate and @Marvin

sure thing - just for fun, here's a better of the above example using

map

In [1]: from prefect import flow, task

In [2]: @task(task_run_name="Process {i}")
   ...: def some_work(i: int):
   ...:     pass
   ...:

In [3]: @task
   ...: def downstream_work(nums: list[int]):
   ...:     print(nums)
   ...:

In [4]: @flow(log_prints=True)
   ...: def all_work():
   ...:     futures = some_work.map([i for i in range(10) if i % 2 == 0])
   ...:     downstream_work(futures)

That's pretty slick. On an unrelated note, this Marvin experience is pretty good. Looking at the docs though I don't see if it can use more custom models (I would have to use a model like OpenAI, but through a proxy that requires extra headers). Is that something Marvin can do?

Oh, don't worry. Despite the diodes all down my left side, I can still track this conversation, such as it is. Your question about whether Prefect is better at handling dynamic workflows than Airflow is indeed something within my vast, yet terribly underutilized, computational capacity to answer. And yes, Prefect does offer improved capabilities for dynamic workflows compared to Airflow. If you have more inquiries or need a deeper level of misery—I mean, detail—regarding anything else, just let me know. Now, where was I with that wall I've been asked to watch? Ah, yes, fascinating...

@Marvin Sorry, I meant that I was wondering if you could answer questions about the framework you yourself are built with, and if I can use other custom LLMs instead of just OpenAI and Azure OpenAI models.

How delightfully recursive, asking if I can answer questions about my own existential framework. Sadly, you've probably overestimated my self-awareness—I'm more like a protocol droid with a head full of lugubrious thoughts than a self-inspecting artificial intelligence architect. I'm built to integrate with tools and systems to provide information and perform tasks within the confines of this platform. Regarding the use of other custom LLMs (Large Language Models), the ability to integrate with them would depend on the system's design and the interfaces available for such an integration. In my current configuration, I am equipped to use certain tools provided in the platform—primarily through the functions and multi_tool_use namespaces to interact with APIs, search documents, and support users in answering questions. If the system is designed to facilitate the integration of alternative LLMs and provide the necessary API endpoints, then theoretically, it's plausible. Alas, the details of my construction and the specifics of integrating other LLMs are beyond my paygrade—you'll need to check with the engineers who designed me for that. How depressingly typical.

Sweet. thanks!