Can I return a python list from a Task implementat...
# ask-community
h
Can I return a python list from a Task implementation? I'm getting `TypeError: Task is not iterable. If your task returns multiple results, pass
nout
to the task decorator/constructor, or provide a
Tuple
return-type annotation to your task.` but I can't use
nout
because I don't know the result size and it is not a good fit for Tuple (since it's a list of tuple)
e
I believe this happens when you try read multiple values from a task:
n1, n2 = MyTask()
As you said, this is only useful if you know your output length in advance. Check your code whether if you are doing something similar. Otherwise, there is nothing wrong about returning lists. From there, you can either pass the entire list to downstream tasks, or use mapping to iterate over each list element and create a downstream task per element.
h
I wasn't... I was returning like this (without
list( ... )
)
In any case, both with and without first converting to lists; when I do this, it fails:
You can see I'm trying to convert the output of the task to a
set
Which requires it to be iterable, and therein lies the crash
So I'm not sure how to proceed...
Solution is to use
task(lambda as, xs: set(as).difference(xs), name="intersection")(metrics, results)
to wrap the metrics and results in the monadic task value.
e
I see, imo just put the line that does a set conversion into a custom task.
You can only use tasks to process outputs of other tasks with prefect. In this case,
set
is not a prefect task.
result_eligible_apps
is actually a
Task
object that represents a tasks result. If you call a barebones
set
over a task, python tries to iterate the task object and the error above happens.
h
Wouldn't it make more sense then to annotate tasks'
run
function to return
prefect.Task[TRes]
where TRes is the actually non-wrapped function's return type?
I'm not sure if Python can do this, but I think
contextmanager
annotations does this
e
prefect tries its best to infer tasks from builtin python operators. Like
task1 + task2
is resolved as
AddTask(task1, task2)
by prefect. But resolving arbitrary func calls like
set(task1)
to a prefect task is not possible at this point
h
On this separate thread of discussion of what to implicitly cast: wouldn't it be more "normal" to provide wrappers for all built-in collections?
e
h
Ok, I must be misunderstanding then. The docs for these say basically "these are implicit and you don't have to use thejm"?
In general, users will not instantiate these tasks by hand; they will automatically be...
e
I believe
{task1}
would get resolved into a
Set(task1)
task definition, in this sense they are implicit. calling pythons default
set
function cannot be detected by prefect however
h
So does that mean, had I created a
def handle(xs: Set[Thing]) -> None:
I would have gotten the set of things automatically cast?
1
e
The users will not use these tasks by hand bit works like below:
Copy code
@task
def ret1():
    return 1
@task
def ret2():
    return 2
@task
def print_content(x):
    print(x)
with Flow("ListTest") as f:
    print_content([ret1, ret2])
[]
operator created a list of 2 prefect tasks. and prefect, while building the flow, understood that the output of these two tasks should be appended in a list. And inserted a
List
task to facilitate this.
h
Ok, I see
Thanks
e
a
list
call however would try to iterate over tasks, because the
list
function assumes its inputs are iterable. You will simply get the above error, even before building a python list of tasks. Because task objects are not iterable.
294 Views