hello prefect community! 🙂 I have a question regarding prefect and using it in combination with a multiprocessing task. I run into an interesting scenario. I was wondering if somebody could explain to me what is happening. If this is the wrong channel please feel free to point me to the correct channel. I have the following setup:

from multiprocessing import Process, active_children
from time import sleep
from prefect import flow, task

def prcoessing_task():
    sleep(3)
    print("finished processing")


def spawning_task():
    print("starting children")
    process = Process(target=prcoessing_task)
    process.daemon = True
    process.start()


def waiting_task():
    processing = True
    while processing:
        for process in active_children():
            print(f"waiting for: {process.pid}")
            process.join()
            processing = False
            print(f"process is done")

@task
def run_processes():
    spawning_task()
    waiting_task()


@flow()
def main() -> None:
    run_processes()


main()

When i run this, my waiting_task waits forever, even though my

processing_task

has finished. The actual process associated with the processing_task does still show up as a running process on

htop

. If I run

run_process

as a normal function (so without decorating it as a task), the flow finishes as expected. Would anybody know what is going on here? Thanks in advance!