Hi, I am trying to implement a geographical data processing worflow, working on datasets that do not fit into memory, and I dont find a proper documented way to perform parallel computation on each object in my datasets. What I would like to do is not return all objects, as the result of a task, to map a function on them. I would like to read data inside a task and dispatch objects (either using a queue or some list containing a reasonnable number of objects) for parallel processing, until I have read and processed all data (+ dumped results into a DB or file, that will be handled by a later task), and I dont find a way to do so with Prefect. It seems I need to return all objects, which I cant, in order to parallelize computation on them with the map function. I have tried using the executor inside a task to call its map function, which works fine and could suit my needs. But I am not sure if this is the right way to do so, and if it can be considered as an official feature you will keep supporting. According to you, what is the best solution to handle this issue ?

Hi @Bertrand GERVAIS, taking a look now! Will be back to you with a recommendation ASAP.

Could you instead return a list of references to the data? For example, if you have 1000 files, you don't need to read them all and return the results, you can map over the list of file names.

I want to avoid returning a list of references to the data, as I am using a generic data reader (OGR library) and my source data can be either local files, DB or served by web services. OGR gives me a good abstraction to read this kind of data transparently, that I dont want to lose by using Prefect

I want to avoid returning a list of references to the data, as I am using a generic data reader (OGR library) and my source data can be either local files, DB or served by web services. OGR gives me a good abstraction to read this kind of data transparently, that I dont want to lose by using Prefect

Clarifying, just to ensure we're not talking past each other. Frequently when I see workloads like this, operations can be broken down into two steps: • List references to all the objects you want to process (usually fast, doesn't require reading the data). This may be a list of files, a list of keys in the db, etc... • Do some work on each object. This task would take the reference, and use the underlying library (OGR) to read the data in and process it. If your workload fits, this, then prefect's map should work:

@task
def items_to_process():
    return list_of_keys_to_process

@task
def process_single_item(key):
    data = read_in_data_from_key(key)
    do_some_work(data)

Thanks for the additional information. I dont think listing references would suit my needs, as I basically have very few files encoded as GeoPackage, which is basically a SQLite DB, or sometimes PostgreSQL tables, and I want to split processing on a table. I am not not very keen on accessing this DB from many workers so that each on of them works on some items in the table. Do you maybe have a better solution for this ?