Hello guys, this should be fairly easy but i dont know the most "Prefectish" way of doing it. Imagine we have a Task do_something_big that is too big and we want to separate it into a few different methods. Is it possible to have a main Task that calls other tasks inside it (that are apt for dask's parallelism)? My first approach of having them all in the flow definition as flow_part_1, flow_part_2, flow_part_3 (depending on eachother) works but I'd rather have a much simpler gantt chart being generated out of it with 2 or 3 (at most) main Tasks. As an example:

@task
def i_am_too_big():
"""
multiple
 lines of code
 that should
 go into 
a different method
"""

"""
some
sequential (to be parallelized)
http
calls
""""

"""
multiple
 lines of code
 that should
 go into 
a different method
"""

Hi @bruno.corucho - not completely sure if this is helpful but is there any reason you couldn't split the big task into smaller tasks and call them as you would any other task (from within the flow context)? If it's purely a matter of code organization and you'd prefer that the success/failure/etc of that big task stay under the same umbrella, you can always subclass Task and take advantage of class methods, something like this:

from prefect import Task

def MyLargeTask(Task):
  def small_method_1(self):
    # do something

  def small_method_2(self):
    # do something

  def run(self):
    # do something

Hello @nicholas, that would be my intended/ideal setup, actually! Our previous run gave us a very odd gantt chart that brought us some questions:

I agree with @nicholas overall, but just to follow down the other track, seeing your post made me think of this convo in this slack thread about using dask inside Prefect tasks so just wanted to forward it for your perusal (tl;dr you can if you wan’t, be careful about returning dask futures though): https://prefect-community.slack.com/archives/CL09KU1K7/p1594661552064000

Untitled.txt

Hi @bruno.corucho - you won't be able to use

.map

because that's a task-level operator; you're attempting to use it on results returned from normal python methods. Each of those functions you've defined in your subclass is a regular python function and so should be treated as such, it's the

PeanutsTask

as a whole that can be operated on as a Prefect task

@nicholas gotcha. This means i can still have a result being returned from *PeanutsTask(*run) that can then be used as a dependency for a future task, just like any other normal Task, correct? Thanks in advance

That's correct!