Hey P team, a question about migrating to RunConfigs:
We’re on Prefect 0.14.4 and looking to migrate off of LocalEnvironment (also using DaskExecutor + FargateAgent). We are currently relying on LocalEnvironment’s
on_start
arg in order to bootstrap django when our flow runs. I’m not seeing any replacement for that in the new RunConfig world. Did I miss something… or is there another pattern for this? Thanks!
a
Amanda Wee
01/26/2021, 9:25 PM
Would it be possible to do that bootstrapping in a state handler that handles the transition of the flow from pending to running states? I'm doing such startup for a shared database connection (two, actually) that gets reused across multiple tasks, and it seems to work, but I don't know if it'll work for django.
👍 1
j
Jim Crist-Harif
01/26/2021, 9:57 PM
on_start
hooks haven't been added back (yet). The hope was other patterns might suffice.
In your case, it sounds like your tasks rely on some implicit local state - if this fails to set up properly you probably want to be informed of failure (and get logs), just like any other tasks. You might find the
We're open to adding `on_start`/`on_stop` hooks back, but want to make sure that what they're being used for can't be better done using other methods. Please let me know if the
resource_manager
option isn't a good fit for your needs.
j
Jeff Brainerd
02/05/2021, 1:35 AM
Thanks @Jim Crist-Harif for the context & info. I will give the resource manager a shot.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.