https://prefect.io logo
h

haven

08/04/2021, 6:16 AM
Hi guys, I'm looking to build a
class MyFlowRunner(prefect.engine.flow_runner.FlowRunner)
(or subclassing another component - open to suggestions!) inside a custom package
my-prefect
to send some HTTP requests before a
Flow
starts, and after it ends. My questions are: • where would be the best way for me to implement such a functionality? along with the considerations: • keep the Prefect API to be as pure as possible • allow the implementation of the
MyFlowRunner
(or some other custom component) to be uncoupled, i.e. if a project upgrades
my-prefect
, the flow configuration doesn't have to change at all Would appreciate comments/suggestions/discussions from whoever are interested/have had experience doing so!
k

Kevin Kho

08/04/2021, 2:28 PM
Hey @haven, could you give me some clue to what those HTTP requests are such that they can’t be done inside the Flow?
h

haven

08/04/2021, 3:34 PM
The HTTP requests I'm trying to make is a metadata submission sort of request. The idea is to allow thousands of applications/flows to have such submission without modifying all these applications manually.
k

Kevin Kho

08/04/2021, 3:40 PM
Ok I think we can subclass the
Flow
class instead. Will try to make an example.
h

haven

08/04/2021, 3:44 PM
That's awesome!
k

Kevin Kho

08/04/2021, 6:19 PM
Ok so I talked to the team and the best way to do this is actually with a state handler attached to the Flow. If you use a state handler, you can run code whenever the code Flow goes from a scheduled to running state. In that state handler you could send you queries. And then just attach the state handler to the flow level. This will be a lot easier than subclassing the FlowRunner or FLow.
h

haven

08/05/2021, 5:44 AM
Ah that sounds neat. Let me try that out! Thanks @Kevin Kho!
5 Views