https://prefect.io logo
Title
a

Aakarsh Nadella

09/05/2019, 8:53 PM
Hello Everyone, I have 2 questions. 1. Can we have more than 1 DAGs running concurrently in the same environment ? 2. Can we have executing different instances of same DAG running concurrently, with each instance having different parameters ?
c

Chris White

09/05/2019, 9:16 PM
Hi @Aakarsh Nadella: 1. depends on what you mean by “environment”; you can of course call
flow.run()
in as many different places / processes as you’d like 2. similar to the above, you can call
flow.run(parameters={})
in many parallel python processes without any issues I suspect you’re asking a deeper question about orchestration of workflows / kicking off flow runs asynchronously via some sort of API, etc. in which case I’d recommend you take a look at Prefect Cloud (https://docs.prefect.io/cloud/faq.html), but let me know if I am misunderstanding your questions!
a

Aakarsh Nadella

09/05/2019, 9:58 PM
Hi @Chris White: So you mean we can run "n" number of flows of same DAG and each flow with different set of parameters ?
c

Chris White

09/05/2019, 10:01 PM
yea sure! Unless I’m missing some nuance of your question, this would work:
python my_flow_file.py &
python my_flow_file.py &
python my_flow_file.py &
...
and you could parse command line arguments as parameter values if you’d like
a

Aakarsh Nadella

09/05/2019, 10:06 PM
Gotcha ! But can we run all the command line arguments mentioned above at once ? Looks like when we enter the the first argument, it starts execution and we can execute the second argument (with different parameters) only after the first one completes
c

Chris White

09/05/2019, 10:08 PM
Yea the
&
should run each process in the background, but that can be platform specific (you might need to use
&&
,
nohup
or
screen
to background the processes, depending on your OS)
a

Aakarsh Nadella

09/05/2019, 10:12 PM
Nice. Thanks for the information. Could you please share me the documentation of using
&
for executing concurrent processes
a

Aakarsh Nadella

09/05/2019, 10:14 PM
Thanks Chris.
👍 1