Hey I’m passing a fairly large object from one task and then map a following task over an iterable with that big object being unmapped.

Consider scattering large objects ahead of time
with client.scatter to reduce scheduler burden and 
keep data on workers

    future = client.submit(func, big_data)    # bad

    big_future = client.scatter(big_data)     # good
    future = client.submit(func, big_future)  # good

I was wondering, what would be the prefect pattern here to scatter the object ahead of time?

Even if it worked, it’s not best practice since your client/scheduler needs to upload this large object to each of your workers. It would be better if this object were somewhere like S3 and the workers i independently pulled it from there. Also, scatter specifically fails with autoscaling

Thanks for the reply.

Yes. That way, it’s the worker doing the work