m

    Matt Wong-Kemp

    2 years ago
    Before I start down a blind alley, how does serialization/Results interact with Resources? Do my resources need to be pickle-able?
    Specifically, the thing I'm thinking about is using a resource manager to manage a dask actor, so I guess the behaviour I'm looking for is: 1. Init/setup is re-run if the flow is restarted 2. Probably no serialization Alternately can I use the resource manager like an actor?
    Jeremiah

    Jeremiah

    2 years ago
    Matt - very interesting. In general, the resource manager is a helper for setting up task dependencies that are commonly used but difficult to configure (appropriately setting triggers for setup and tear down tasks, for example) - but the key is that each piece is indeed a task (and subject to normal task rules regarding serialization). You might want to keep an eye on https://github.com/PrefectHQ/prefect/pull/3139 for true object reuse in a shared memory space
    I suppose you could use it to interact with an actor, essentially treating it as an opportunity to create “pre/post flow hooks” - perhaps Jim has a more in depth opinion
    m

    Matt Wong-Kemp

    2 years ago
    awesome, this looks like what I want. One slight hiccup is whether this one one-per-process or one-per-flow - one-per-process will do for my use-case, I'm basically just looking to share some mutable work between tasks (and Actor looked like the dask idiom to do that, but this is a Prefect idiom)