https://prefect.io logo
Title
e

Enrique

07/13/2022, 10:39 AM
Hi all! 🙂 I have a question (maybe it's stupid but here it goes xD) I'm working with Prefect in a Kubernetes environment, which means that the agent and the jobs are running in the cluster. The cluster has 2 CPUs. When I'm trying to run three parallel tasks, only two of them run and then when one of them ends the third one starts running. Is that the expected behavior or I'm doing something wrong? Thanks a lot for your tool, Prefect rocks! 😎
😎 1
1
a

Anna Geller

07/13/2022, 10:52 AM
Can you share your flow code? by default, a ConcurrentTaskRunner is used which will run your tasks concurrently (IO operations), but won't parallelize work across CPUs
e

Enrique

07/13/2022, 11:07 AM
...
# task 1
set_status_task_1_processing = update_status(
merge_processing_backend, constants.TASK_1
)
task_1_processed = process_task_1(
set_status_task_1_processing
)
set_status_task_1_finished = update_status(
task_1_processed, constants.STATUS_TASK_1_FINISHED
)
# optional task 2
with case(task_2_needed, True):
set_status_task_2_processing = update_status(
merge_processing_backend, constants.TASK_2
)
task_2_processed = process_task_2(
set_status_task_2_processing
)
set_status_task_2_finished = update_status(
task_2_processed, constants.STATUS_TASK_2_FINISHED
)
# optional task 3
with case(task_3_needed, True):
set_status_task_3_processing = update_status(
merge_processing_backend, constants.TASK_3
)
task_3_processed = process_task_3(
set_status_task_3_processing
)
set_status_task_3_finished = update_status(
task_3_processed, constants.STATUS_TASK_3_FINISHED
)
merge_end = merge(
set_status_task_1_finished,
set_status_task_2_finished,
set_status_task_3_finished,
)
what it's currently happening is that if the task 2 and 3 are required they run in parallel but task 1 keeps waiting until one of the other two ends 😅
a

Anna Geller

07/13/2022, 11:15 AM
you can disable sending all responses to the main channel by not checking this checkbox - could you fix that to keep the main channel clean?
:upvote: 1
:thank-you: 1
e

Enrique

07/13/2022, 11:16 AM
ups, sorry! 😅
a

Anna Geller

07/13/2022, 11:29 AM
no worries, happens to me every now and then too - thanks so much! so you have some quite complex task dependencies with branching logic and this is not well suited for parallel processing - since you're using 1.0, there is a concept of mapping to run things in parallel - can you check if using mapping helps? https://docs.prefect.io/core/concepts/mapping.html
e

Enrique

07/13/2022, 12:28 PM
Thanks a lot! 🙂 I'm going to try the mapping then!
🙌 1