haven

    haven

    1 year ago
    Hi guys, I'm looking to build a
    class MyFlowRunner(prefect.engine.flow_runner.FlowRunner)
    (or subclassing another component - open to suggestions!) inside a custom package
    my-prefect
    to send some HTTP requests before a
    Flow
    starts, and after it ends. My questions are: • where would be the best way for me to implement such a functionality? along with the considerations: • keep the Prefect API to be as pure as possible • allow the implementation of the
    MyFlowRunner
    (or some other custom component) to be uncoupled, i.e. if a project upgrades
    my-prefect
    , the flow configuration doesn't have to change at all Would appreciate comments/suggestions/discussions from whoever are interested/have had experience doing so!
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @haven, could you give me some clue to what those HTTP requests are such that they can’t be done inside the Flow?
    haven

    haven

    1 year ago
    The HTTP requests I'm trying to make is a metadata submission sort of request. The idea is to allow thousands of applications/flows to have such submission without modifying all these applications manually.
    Kevin Kho

    Kevin Kho

    1 year ago
    Ok I think we can subclass the
    Flow
    class instead. Will try to make an example.
    haven

    haven

    1 year ago
    That's awesome!
    Kevin Kho

    Kevin Kho

    1 year ago
    Ok so I talked to the team and the best way to do this is actually with a state handler attached to the Flow. If you use a state handler, you can run code whenever the code Flow goes from a scheduled to running state. In that state handler you could send you queries. And then just attach the state handler to the flow level. This will be a lot easier than subclassing the FlowRunner or FLow.
    haven

    haven

    1 year ago
    Ah that sounds neat. Let me try that out! Thanks @Kevin Kho!