Hey guys, I'm still new to prefect, and trying to figure it out. I'm testing out a simple task. I have a couple dozen sql files, each sql file contains mutiple sql statements I want all the files to be executed against the database at once in parallel. I tried multiple different ways, using the dask executor example, and now the using applied_map. No matter what I seem to do, on prefect cloud, everything runs one by one. what am I missing? my current attempt

files = []
for file in os.listdir(path):
    files.append(path+file)

@task
def run_file(file):
    with open(file) as f:
        line = f.readline()
        name = f.name
        sql = ""
        for _, line in enumerate(f):
            sql += line
            if line.rstrip().endswith(";"):
                query.run(query=sql)
                sql = ""

with Flow("parallelfiles") as flow:
    apply_map(run_file, files)

flow.register(project_name="test")

Hye @ARun are you setting the Dask executor in your flow’s environment like the examples here https://docs.prefect.io/orchestration/execution/local_environment.html#examples?

Thanks! That's what I needed. In the flow run schematic, is there a way to show all the mapped runs at once instead of just the task? For Example, this is what I currently see. Instead of seeing just run_file, is there a way of having it branch out to all the mapped runs?

I believe these visualizations are still undergoing development and currently it doesn’t have an easy way to fan out the runs besides scrolling through the mapped children. I’m pretty sure this enhancement is on the docket for future improvements 🙂

sounds good, thanks @josh Can we dynamically name tasks as we call them?

Dynamic names for mapped tasks is currently proposed on the repo here https://github.com/PrefectHQ/prefect/issues/2100 however in the initial implementation there was not a good spot for it to live in the mapped pipeline so it is still up for debate. Feel free to voice your support / need for the feature on that issue!