I appreciate the references, will peep

@Nate, fwiw, I was able to get really really close to what I was looking for via the code snippet below

@flow(task_runner=ThreadPoolTaskRunner(max_workers=5), log_prints=True)
def job():

    result = schema_setup()
    collect_result = collect()
    post_ids = collect_result.get("post_ids")
    print(f"Posts to process: {len(post_ids)}")

    if len(post_ids) > 0:
        print("starting map operation over the identifed posts")
        processed_posts = []
        for post_id in post_ids:
            processed_posts.append(ingest.submit({"post_id": post_id}))
        wait(processed_posts)
    else:
        print("No posts to complete, exiting now.")

In my case, the ingest task doesn't return anything (its doing data pipeline work in a Google Cloud Function). I borrowed everything from here in the docs. The retries on the task did what I wanted on the error front and seeing the tasks spawn as others completed is exactly what I wanted. My question: I am not really wrapping my head around wait. I know that I am submitting tasks to the threadpool, but not certain what

wait(processed_posts)

is really doing, as I borrowed that from the floors example. I get the idea I am submitting the job to the threadpool, but do I need to use the list and

wait(processed_posts

or can I just call

ingest.submit({"post_id": post_id})

and move on?

Unrelated, holy moly it is refreshing to see someone hop into an ipython terminal. hattip

i’m a big fan for quick scripting! it’s also good when trying to communicate/teach bc it’s hard to hide implicit context, “no sleight of hand!”