https://prefect.io logo
Join Slack
Powered by
Hey, i'm starting to play with prefect Orion for a...
# prefect-community
f
Hey, i'm starting to play with prefect Orion for a project and i was wondering how dynamic the flows can be ? For this project i would need to generate a flow dynamically depending on a json config. The tasks would be static but the execution of those tasks can be very different from a run to an other. For example, i have an extract task. A flow could be the execution of this extract task once or could be this extract class followed by the same extract task with different parameters and using the output of the first one. I started to play with for loops but it seems that it is made to execute tasks in a parallel manner. I haven't found how to set execution plan when using a for loop.
a
To run things in parallel you can use the default ConcurrentTaskRunner or Dask- or RayTaskRunner Re dynamicism: think of it this way - you point Prefect at your flow file e.g. in S3 or GitHub and every time your flow runs, it THEN gets evaluated, so every time it runs it could have different tasks and could be versioned using a different flow version - all without having to recreate the deployment or reregister your flow (using 1.0 terminology)
f
Ok thanks for your answer. Regarding the dynamic possibilities, how would you build a flow that can dinamically have different shapes based on an input config ? I tried to set dependencies between tasks instanciated in a for loop but i could'nt find how
managed to do it by assigning a task in a list and using this list in to loop like this:
Copy code
@flow
def run_print(nb):
  task_list = []
  for i in range(nb):
          if i != 0:
              my_task = print_nb(i, wait_for=[task_list[i-1]])
              task_list.append(my_task)
          else :
              my_task = print_nb(i)
              task_list.append(my_task)
a
nice you figured it out! 🙌
7 Views
Open in Slack
PreviousNext
Powered by