Chris Martin
09/04/2020, 5:38 PM@resource_manager
class DaskCluster:
def init(self, n_workers):
self.n_workers = n_workers
def setup(self):
return Client(n_workers=self.n_workers)
def cleanup(self, client):
client.close()
In this case how does the Client
get passed to tasks? Is is pickled and sent around? Moreover if the client held some mutable state that could be updated by the tasks, what would happen?nicholas
09/04/2020, 5:49 PMwith/as
syntax, like this:
with Flow("My Flow") as flow:
with DaskCluster(n_workers=2) as client:
my_task(client)
To learn more about how the Client works, I'd encourage you to check out the code in the core tasks: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/tasks/core/resource_manager.pyChris Martin
09/04/2020, 5:52 PM