I'm having a problem creating a prefect flow to process a large number of xml files (40k) without running out of ram. Is this the right place to ask for help?

Hi @Jon Ruhnke, here is the right place for help. Are you using LocalDaskExecutor or DaskExecutor?

I don't have a clue. This is my first prefect flow, and i've written most of it already and it seems to work pretty well on smaller file sizes. I'm just running it via command line

Ah ok. If you didn’t set any it would just be the local sequential executor. I think this will help you. If you are dealing with large data, you can reduce your footprint by writing the results somewhere, returning the location, and then reading it back in later.

I spent a bunch of time implementing mapping so that my ETL task would run in batches of 1000 files, thought that was going to solve my problem, but it doesn't seem like that affects memory usage?

It does a bit. What is the return of your task?

It won’t because the result is held in memory to pass it to the next task. If

parse_file

normally succeeds, you can reduce your footprint by combining these tasks. If not, you can just save that dataframe, return the location and load it in the

load_to_sql

task.

By "save that dataframe", you mean save it to a file on disk like json or something?

Yes exactly

Otherwise I could combine parse/load into a single function so no data is being held? That seems like a bad practice to combine a bunch of thing into a single task

Yes you are exactly right. Combining into the single task doesn’t mean combining the code. You can still keep it as a function and call it inside a task

def some_function():
    return 1

@task
def some_task():
    return some_function() + 1

If an operation is pretty much guaranteed to succeed, you don’t need to put it in a task since you don’t need observability and retries around it.

Oh that makes sense

No problem!