@Marvin is it viable to use the subprocess module to call sub-flows and tasks? I have a scenario where the code runs, but Prefect does not pick up the flows and tasks called in the subprocess call for

python

,

script.py

So, you want to know if it's a good idea to use the

subprocess

module to call sub-flows and tasks in Prefect. It sounds like you're trying to give Prefect the silent treatment, and then you wonder why it's ignoring you. Before we continue down this vibrant path of inventive programming techniques, I need to know which version of Prefect you are using. Is it 2.x or 3.x? This helps me tailor my deeply enthusiastic search for your answer.

@Marvin Prefect 2. However, it's not that we want the silent treatment. The subprocess module does run a python script decorated with prefect @flow and @task methods

Got it, you want to have your cake and eat it too—running your decorated flows and tasks in a subprocess and expecting Prefect to pick up on them. Let me search the Prefect 2.x documentation to see if there's any guidance or related information on this rather unique approach. One moment. Ah, so close yet infinitely far. It appears the documentation does not explicitly cover launching flows and tasks from a subprocess. It does cover subflows and tasks, but that's like comparing apples to oranges, except less delicious. Here are some key points from what I found: 1. Subflows: Prefect treats subflows like normal flow runs, each with a full representation in the backend. A new task runner is created for the subflow, and it blocks execution of the parent flow until it is complete. 2. Tasks vs. Subflows: Prefect recommends using tasks for discrete, specific pieces of work and subflows when organizing more complex workflows. Links to the sources, so you can drown in the minutia: - Concepts of Flows and Subflows From the information, it seems that Prefect expects tasks and flows to be managed within its own execution context, not within a subprocess where it loses visibility. If you want to stick with subprocesses, there's always good old logging to the rescue and manually tracking status, but that does seem quite contrary to Prefect's design. If you're committed to this adventure, I would suggest peeking at GitHub issues to see if anyone else has tread this arduous path. Shall I proceed with that?

hi @Joshua Caskie - im curious about your scenario can you share an example?