Our flow feels pretty simple and looks something like this :

batch_size = Parameter("batch_size", default = 50)

codes = get_codes() # Gets a list of codes from crawling a website (~13 000 codes)
batches = batch_generator(codes, batch_size) # Generates batches in order to parallelize tasks
documents = get_documents.map(batches) # Loads a document for each code in the batch. Runs about 30s to 2 min for a batch of size 50
uploaded_documents = upload_documents(documents) # Uploads the loaded documents to a DB. Runs ~5 min to 20 min for a batch of size 50
parsed_documents = parse_documents.map(uploaded_documents) # Parses the uploaded documents using a home API. Runs about ~1 min for a batch of 50.

We run this Flow on a DaskExecutor with 2 workers. The current configuration generates maps of ~300 childs The issue we see is the following : • The first tasks run flawlessly as expected • The first upload tasks run correctly • At some point, while 4 upload tasks are already running, every remaining pending child upload task are launched at the same time with a

None

input value, which makes them fail. • The 4 tasks that were running at this time end up being killed by the Zombie Killer after a while. • As we are using a Dask Executor, DFE allows a portion of downstream parsing tasks to succeed. The rest is divided between failed tasks (receiving

None

inputs as well) and TriggerFailed tasks.

Heres the schematic of our last run. There is an additional parameter but it is not relevant whatsoever to our issue.

For details, 51/311 upload runs succeeded, 4 were killed by the Zombie Killer and the rest were launched with a

None

input and failed because of it. 51/311 parsing tasks succeeded, 225/311 failed and 35/311 TriggerFailed

If at all relevant, it took about 2.5h between launching the first upload job and the "miss-launched" ones.

It allows us to still parallelize while not creating a child for each element of the initial list which is approx. 13K elements long

Is the

uploaded_documents = upload_documents(documents) # Uploads the loaded documents to a DB. Runs ~5 min to 20 min for a batch of size 50

a mapped operation?

I am wondering if this fails due to DB concurrency?

Do you know if your DaskExecutor workers are dying/restarting?

I have no idea about the DaskExecutor workers dying, I don't know how I could get information about it

We are using CouchDB, concurrency does not seem great but since ~50 upload jobs succeed before the flow going crazy, I don't think the issue is directly linked to it but I could be wrong.

Just relaunched it with a 500 documents limit in 10 batches, I'll keep you posted when I get the result just in case.

The "small scale" run did succeed.

Is there any way that running longer and doing more stuff makes the executors run out of memory or something ?