Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi all! :slightly_smiling_face: I have a question (maybe it's stupid but here it goes xD) I'm working with Prefect in a Kubernetes environment, which means that the agent and the jobs are running in the cluster. The cluster has 2 CPUs. When I'm trying to run three parallel tasks, only two of them run and then when one of them ends the third one starts running. Is that the expected behavior or I'm doing something wrong? Thanks a lot for your tool, Prefect rocks! :sunglasses:

Can you share your flow code? by default, a ConcurrentTaskRunner is used which will run your tasks concurrently (IO operations), but won't parallelize work across CPUs

`...`

`# task 1`
`set_status_task_1_processing = update_status(`
    `merge_processing_backend, constants.TASK_1`
`)`
`task_1_processed = process_task_1(`
    `set_status_task_1_processing`
`)`
`set_status_task_1_finished = update_status(`
    `task_1_processed, constants.STATUS_TASK_1_FINISHED`
`)`

`# optional task 2`
`with case(task_2_needed, True):`
    `set_status_task_2_processing = update_status(`
        `merge_processing_backend, constants.TASK_2`
    `)`
    `task_2_processed = process_task_2(`
        `set_status_task_2_processing`
    `)`
    `set_status_task_2_finished = update_status(`
        `task_2_processed, constants.STATUS_TASK_2_FINISHED`
    `)`

`# optional task 3`
`with case(task_3_needed, True):`
    `set_status_task_3_processing = update_status(`
        `merge_processing_backend, constants.TASK_3`
    `)`
    `task_3_processed = process_task_3(`
        `set_status_task_3_processing`
    `)`
    `set_status_task_3_finished = update_status(`
        `task_3_processed, constants.STATUS_TASK_3_FINISHED`
    `)`

`merge_end = merge(`
    `set_status_task_1_finished,`
    `set_status_task_2_finished,`
    `set_status_task_3_finished,`
`)`

what it's currently happening is that if the task 2 and 3 are required they run in parallel but task 1 keeps waiting until one of the other two ends :sweat_smile:

image.png

you can disable sending all responses to the main channel by not checking this checkbox - could you fix that to keep the main channel clean?

no worries, happens to me every now and then too - thanks so much!

so you have some quite complex task dependencies with branching logic and this is not well suited for parallel processing - since you're using 1.0, there is a concept of mapping to run things in parallel - can you check if using mapping helps? <https://docs.prefect.io/core/concepts/mapping.html>

Thanks a lot! :slightly_smiling_face: I'm going to try the mapping then!