hi @Dylan, how would I go setting this up?
So here are the steps I believe will have to follow:
1. Create prefect_shell_script.py which will call the .sh file (that does the spark-submit command) via ShellTasks or possibly just call all the commands the .sh file does but using ShellTasks (basically omitting the .sh and going straight to the commands). Either way of submitting the spark-submit request will use a python file named spark_job.py
2. At the end of the prefect_shell_script.py register all the tasks in thy py file to be a flow
3. In the code of spark_job.py it should contain prefect code (task and code to register the flow as well)
So I know call the flow registered to run the prefect_shell_script.py code (lets call it flow A) but how will the flow that was registered for the spark_job.py file (flow B) be fired?
Flow A does the spark-submit but it calls a python code, how would the Flow B be triggered it has not been specifically requested? I will assume in the UI it will only Flow A appear as running but Flow will not.
thanks!