Matt Wong-Kemp

09/03/2020, 11:02 PM
Before I start down a blind alley, how does serialization/Results interact with Resources? Do my resources need to be pickle-able?
Specifically, the thing I'm thinking about is using a resource manager to manage a dask actor, so I guess the behaviour I'm looking for is: 1. Init/setup is re-run if the flow is restarted 2. Probably no serialization Alternately can I use the resource manager like an actor?


09/03/2020, 11:19 PM
Matt - very interesting. In general, the resource manager is a helper for setting up task dependencies that are commonly used but difficult to configure (appropriately setting triggers for setup and tear down tasks, for example) - but the key is that each piece is indeed a task (and subject to normal task rules regarding serialization). You might want to keep an eye on for true object reuse in a shared memory space
I suppose you could use it to interact with an actor, essentially treating it as an opportunity to create “pre/post flow hooks” - perhaps Jim has a more in depth opinion

Matt Wong-Kemp

09/04/2020, 8:03 AM
awesome, this looks like what I want. One slight hiccup is whether this one one-per-process or one-per-flow - one-per-process will do for my use-case, I'm basically just looking to share some mutable work between tasks (and Actor looked like the dask idiom to do that, but this is a Prefect idiom)