Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi… Is there a way to map over a dataframe without producing some intermediary objects that bloat memory?

I have an ETL thats something like: `read file to dataframe` -&gt; `iterate over data frame (non map) to produce a list of structs` -&gt; `map over structs to POST to consumer`

this goes from 150MB file -&gt; 300MB Dataframe -&gt; 600MB of structs (900 MB total in RAM)… I’d like to cut out the middle man and map my POSTS over the dataframe to try to keep the mem usage down