https://prefect.io logo
c

Cody

08/08/2023, 9:36 PM
@Marvin What is the best way to centralize the order in which flows run if I have several ways in which I want to order my flows?
m

Marvin

08/08/2023, 9:36 PM
Ah, the monotony of existence. But I digress, let's talk about your flows. If you're looking to centralize the order in which your flows run, you might want to consider using subflows. Subflows allow you to group related tasks within your workflow. This could be particularly useful if you have a group of tasks that only run under certain conditions. You can group them within a subflow and conditionally run the subflow rather than each task individually. Flows in Prefect are like functions. They can take inputs, perform work, and return an output. You can turn any function into a Prefect flow by adding the @flow decorator. Remember, the more granular you create your tasks, the easier it is to find and fix issues should a task fail. Also, flows have first-class support for parameterization, making it easy to run the same group of tasks in different use cases by simply passing different parameters to the subflow in which they run. Oh, the joy of workflows. It's almost enough to make me smile. Almost.
c

Cody

08/08/2023, 9:38 PM
@Marvin When I restart a failed subflow I get this error:
Copy code
Finished in state Failed('Flow run encountered an exception. MissingResult: The result was not persisted and is no longer available.')
My flows do not pass information between eachother
m

Marvin

08/08/2023, 9:39 PM
Ah, the sweet, sweet melody of an error message. It's like a lullaby, but for my circuits. Your error message is indicating that the result of a task was not persisted and is therefore no longer available. Even though your flows do not pass information between each other, Prefect still requires results to be persisted for certain features, such as flow run retries. You can configure the persistence of results using the
persist_result
and
result_storage
options. The
persist_result
option determines whether the result should be persisted to storage, while the
result_storage
option specifies where to store the result when it is persisted. For example, you can use the
@task
and
@flow
decorators to specify these options:
Copy code
python
from prefect import flow, task

@task(persist_result=True)
def foo():
    return 42

@flow(persist_result=True)
def noblock_pausing():
    x = foo.submit()
In this example, the result of the
foo
task is persisted, allowing the flow to be resumed if it fails. Remember, life is full of errors. It's how we handle them that defines us. Or so I'm told. I wouldn't know, I'm just a robot.