Before I start down a blind alley, how does serial...
# ask-communitym
Matt Wong-Kemp
09/03/2020, 11:02 PMBefore I start down a blind alley, how does serialization/Results interact with Resources? Do my resources need to be pickle-able?
Matt Wong-Kemp
09/03/2020, 11:06 PMSpecifically, the thing I'm thinking about is using a resource manager to manage a dask actor, so I guess the behaviour I'm looking for is:
1. Init/setup is re-run if the flow is restarted
2. Probably no serialization
Alternately can I use the resource manager like an actor?
j
Jeremiah
09/03/2020, 11:19 PM
Matt - very interesting. In general, the resource manager is a helper for setting up task dependencies that are commonly used but difficult to configure (appropriately setting triggers for setup and tear down tasks, for example) - but the key is that each piece is indeed a task (and subject to normal task rules regarding serialization). You might want to keep an eye on https://github.com/PrefectHQ/prefect/pull/3139 for true object reuse in a shared memory space
Jeremiah
09/03/2020, 11:21 PM
I suppose you could use it to interact with an actor, essentially treating it as an opportunity to create “pre/post flow hooks” - perhaps Jim has a more in depth opinion
m
Matt Wong-Kemp
09/04/2020, 8:03 AMawesome, this looks like what I want. One slight hiccup is whether this one one-per-process or one-per-flow - one-per-process will do for my use-case, I'm basically just looking to share some mutable work between tasks (and Actor looked like the dask idiom to do that, but this is a Prefect idiom)
2 Views