Hi all, I have a flow in which the first task grabs a set of files (where the number of files can vary) , and then the next task maps over the file locations returned by the first task to submit a batch job for each file. However, i have multiple batch jobs that should be submitted consecutively per file, and i want the first batch job to complete for each file before the second job starts. However, I don't want to wait for all the batch jobs from each round to be completed before submitting the next round of batch jobs (i.e. if the first batch job completes for file_A, i want to go ahead and submit the next batch job for file_A. Is there a way of doing this inside the prefect flow? i could set this up outside of the prefect flow, but then i think i would have to re-register a new flow every time a new set of files is uploaded to s3. Thanks in advance!

Hey @Kathryn Klarich, I think the approach here is to package all of your consecutive batch jobs into a Flow. Then use a main Flow to schedule the batch jobs for each of those independently with

StartFlowRun

. Does this make sense? I have a code example

Thanks! I think it makes sense but am new to prefect so probably need to try it out to make sure. A code example would be great if it's easy to share. I'll read the docs about StartFlowRun as well.

thanks!