Is there a good heuristic for understanding when to actually expect a performance improvement from converting a task that, say, operates over a list in sequence to a mapped task? I have a task that loops over hundreds of CSV files, parses and transforms the data, and writes the parsed data to an output file (one output file for each input file)...I was anticipating that converting this process to a mapped task and increasing my CPU count (I'm running this on Fargate via ECS, so I upped from 0.5 to 2 vCPUs in this specific case) would help out a lot by parallelizing this file-parsing procedure, but it actually has caused that node in my DAG to take much longer than it used to. I'm wondering if the overhead of this mapped task getting compiled and sent to the LocalDaskExecutor is just significantly outweighing the gains from the parallelization, because each mapped task is so small (the files being parsed are not large), and this would only make sense to do if the files were much, much larger?

I think Dask often does not infer CPU count correctly. Did you pass the number of workers to the LocalDaskExecutor? That might help

ah I didn't explicitly pass the number of workers b/c I assumed from the documentation where it said

By default the number of threads or processes used is equal to the number of cores available.

, so if Fargate is giving it access to 2 vCPUs my assumption was that it would just infer that

num_workers=2

is the appropriate thing to do

Ehhh ideally, but I have seen it not work right for many people. We use a Dask utility under the hood for that 🤷. I just recommend explicitly setting the number of workers.

okay yeah it seems like that's helping a lot, it's still slower than doing it sequentially but I'm wondering if this is just because the files I'm working with are quite small; in my production workload they'd be quite a bit bigger

From experience, there is just an overhead with reading/writing CSV files. Do you have something like

year/month/day.csv

and you are right to read these one at a time?

oh interesting, no this is a very legacy pipeline that I recently moved to prefect that just uses vanilla python, not even doing anything with pandas

Parquet would be so much better if your can, but I think duckdb will be super fast for this operation over vanilla Python (because it’s written in C++) too .

would using DuckDB for this mean that all of the transformation / manipulation of the CSV values that I was doing with python would now need to be converted to SQL? or the Python client's API is such that actually the code changes there would be pretty minimal?

You can load in DuckDB and materialize to Pandas I think