Does Prefect have plugin support? Hi all! I'm working on setting up some new pipelines and figuring out what tools to use. One pattern we have now is that the software team maintains the core ETL, and the data science contributes a few final stages to the pipeline to compute new metrics. The DS team contributions tend to change more rapidly, may not be as robust, and the DS team may want to re-run stages with new algorithms more frequently than the base pipeline run. So it would be nice to have separate versioning/releasing of pipeline stages from the different teams. Is there a way with Prefect to pull some portions of the pipeline from a separate repo or anything like that? (We could also manage this outside of the orchestration framework, like have CD combine portions of the pipeline from different sources, or have two dependent pipelines with one triggering the next.)

Hey @Mark Fickett, I think the approach here may be to use a Flow of Flows setup where the DS team would have a separate Flow that you then invoke by a main Flow?

Thanks! So like this: https://docs.prefect.io/core/idioms/flow-to-flow.html#scheduling-a-flow-of-flows . And the different flows could be in separate repos / maintained by separate teams, but so long as they're registered in the same Prefect instance (?) they can reference each other.

Yep that’s right

You can also define a function that calls tasks which will add them to the flow

# Note the lack of @task decorator for this function

def my_pipeline_steps(input_task):
    x = my_task(input_task)  # `my_task` should be defined with the @task decorator
    y = my_other_task(x)
    return my_other_other_task(y)

with Flow(...) as flow:
    my_param = Parameter("my_param")
    sub_result = my_pipeline_steps(my_param)
    ....

Thanks Michael. Sounds like that makes sense for modularizing a flow w/in one codebase, and then flows-of-flows would be the tool for coordinating among different teams / codebases.