Channels
pacc-oct-11-12-2023
pacc-july-11-12-2023
pacc-sept-20-21-2023
pacc-aug-16-2023
pacc-aug-29-30-2023
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
pacc-sept-26-27-2023
pacc-july-6-2023
intros-test
livestream-chatter
marvin-in-the-wild
data-ecosystem
geo-israel
pacc-june-14-2023
geo-japan
pacc-london-2023
pacc-sept-13-14-2023
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
pacc-july-17-18-2023
introductions
prefect-server
data-tricks-and-tips
prefect-aws
prefect-gcp
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-ui
prefect-kubernetes
pacc-aug-22-23-2023
prefect-community
Powered by
#prefect-kubernetes
Title
l
Luke Segars
08/10/2023, 6:24 PM
Hello all! We're using prefect-ray to run our flow runs on a k8s-based ray cluster. In several places we pass large dataframes into mapped tasks and were running out of object store memory, which we just discovered is an antipattern:
https://docs.ray.io/en/latest/ray-core/patterns/pass-large-arg-by-value.html#anti-pattern-passing-the-same-larg[…]value-repeatedly-harms-performance
question: is there a way to access the ray instance created by prefect-ray to call ray.put() and ray.get() manually? what's the recommended way to do that?
👀 1
2 Views
Post