Samuel Hinton

03/01/2023, 2:22 AM
Hey team! So heres a fun one for you - anonymous/dynamic flows. Heres my use case: 1. We have a bunch of endpoints to request data from and save them to an internal DB 2. The data for these endpoints is available one different schedules 3. Each
that fetches data returns it in the same
format, so our
task is very general. Now what I was wanting to do was something like the following:
for data_task in list_of_data_tasks:
   @flow(name=f"{data_task.__name__} flow", task_runner=dask_task_runner)
   def anonymous_flow():
      data = data_task.submit()
I thought “yeah this is good, it saves tons and tons of duplicated
declarations and duplicated
lines. However, this won’t work because the flow is dynamic, so an agent wont be able to actually find it. What would be the best practise solution for the above?
@Ryan Peden btw this is why I was using the registry in the previous question about duplicate names - reducing code duplication for common flow structures and deployments
Noticed there was a previous discussion about this that went unanswered, so have cross posted it there ( and will update here if anyone is able to chime in on either platform