https://prefect.io
Join Slack
if after running a concurrent task run i want to run another task ( after completion of previous con...
# ask-community
d
if after running a concurrent task run i want to run another task ( after completion of previous concurrent task run) with the output of 1st task in concurrent task run. Is it possible ?
✅ 1
a
the simplest might be to combine those two into one task e.g. when I do "extract & load" step without any transformations in between, I'd typically put them into one task since the sole purpose of such task is to get this data into X target (the target is typically even named after source anyway so observability aspect remains)
d
but what i wanna achieve is task A (array output)-> Map(task B) -> task C(output of 1st run of task B) i want the sequential run as well
a
explore various designs and see what's working best, no right or wrong here what you suggest seems doable by doing a for loop without submit for task C and another for loop with submit for task B
Copy code
# pseudo code
for i in task_a_output:
   task_b.submit(i, ...)

for i in task_a_output:
   task_c(i...)
d
Copy code
def spends_flow(event):
    scheduler_output = run_scheduler(event)
    executor_output = run_executor.map(scheduler_output['body_array'])

this is what im doing right now. for concurrent run 
But i want to have one more task (not map state) after all the run_executor tasks have run

something like 

def spends_flow(event):
    scheduler_output = run_scheduler(event)
    executor_output = run_executor.map(scheduler_output['body_array'])
    run_task_c(executor_output[0])
are you suggesting to loop over the output and use task.submit ? would it still be concurrent ?
🙌 1
a
yes exactly! calling a .submit() on a task submits it to a task runner - the default one is concurrent task runner without .submit() you run things sequentially in a flow run process
d
but before i run task c i want to make sure that all concurrent runs of task b have completed..
a
.submit().wait() then
to wait for completion and then start sth else
d
alright. ill try this out
🙌 1
5 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by