https://prefect.io logo
l

Luke Segars

08/10/2023, 6:24 PM
Hello all! We're using prefect-ray to run our flow runs on a k8s-based ray cluster. In several places we pass large dataframes into mapped tasks and were running out of object store memory, which we just discovered is an antipattern: https://docs.ray.io/en/latest/ray-core/patterns/pass-large-arg-by-value.html#anti-pattern-passing-the-same-larg[…]value-repeatedly-harms-performance question: is there a way to access the ray instance created by prefect-ray to call ray.put() and ray.get() manually? what's the recommended way to do that?
👀 1
2 Views