https://prefect.io logo
Powered by
Title
f

Florian Guily

06/01/2022, 9:25 AM
Hey, i'm starting to play with prefect Orion for a project and i was wondering how dynamic the flows can be ? For this project i would need to generate a flow dynamically depending on a json config. The tasks would be static but the execution of those tasks can be very different from a run to an other. For example, i have an extract task. A flow could be the execution of this extract task once or could be this extract class followed by the same extract task with different parameters and using the output of the first one. I started to play with for loops but it seems that it is made to execute tasks in a parallel manner. I haven't found how to set execution plan when using a for loop.
a

Anna Geller

06/01/2022, 9:38 AM
To run things in parallel you can use the default ConcurrentTaskRunner or Dask- or RayTaskRunner Re dynamicism: think of it this way - you point Prefect at your flow file e.g. in S3 or GitHub and every time your flow runs, it THEN gets evaluated, so every time it runs it could have different tasks and could be versioned using a different flow version - all without having to recreate the deployment or reregister your flow (using 1.0 terminology)
f

Florian Guily

06/01/2022, 12:02 PM
Ok thanks for your answer. Regarding the dynamic possibilities, how would you build a flow that can dinamically have different shapes based on an input config ? I tried to set dependencies between tasks instanciated in a for loop but i could'nt find how
managed to do it by assigning a task in a list and using this list in to loop like this:
@flow
def run_print(nb):
  task_list = []
  for i in range(nb):
          if i != 0:
              my_task = print_nb(i, wait_for=[task_list[i-1]])
              task_list.append(my_task)
          else :
              my_task = print_nb(i)
              task_list.append(my_task)
a

Anna Geller

06/01/2022, 1:54 PM
nice you figured it out! 🙌
3 Views
#prefect-communityJoin Slack
Powered by