Thread
#prefect-community
    haf

    haf

    1 year ago
    Can I return a python list from a Task implementation? I'm getting `TypeError: Task is not iterable. If your task returns multiple results, pass
    nout
    to the task decorator/constructor, or provide a
    Tuple
    return-type annotation to your task.` but I can't use
    nout
    because I don't know the result size and it is not a good fit for Tuple (since it's a list of tuple)
    emre

    emre

    1 year ago
    I believe this happens when you try read multiple values from a task:
    n1, n2 = MyTask()
    As you said, this is only useful if you know your output length in advance. Check your code whether if you are doing something similar. Otherwise, there is nothing wrong about returning lists. From there, you can either pass the entire list to downstream tasks, or use mapping to iterate over each list element and create a downstream task per element.
    haf

    haf

    1 year ago
    I wasn't... I was returning like this (without
    list( ... )
    )
    In any case, both with and without first converting to lists; when I do this, it fails:
    You can see I'm trying to convert the output of the task to a
    set
    Which requires it to be iterable, and therein lies the crash
    So I'm not sure how to proceed...
    Solution is to use
    task(lambda as, xs: set(as).difference(xs), name="intersection")(metrics, results)
    to wrap the metrics and results in the monadic task value.
    emre

    emre

    1 year ago
    I see, imo just put the line that does a set conversion into a custom task.
    You can only use tasks to process outputs of other tasks with prefect. In this case,
    set
    is not a prefect task.
    result_eligible_apps
    is actually a
    Task
    object that represents a tasks result. If you call a barebones
    set
    over a task, python tries to iterate the task object and the error above happens.
    haf

    haf

    1 year ago
    Wouldn't it make more sense then to annotate tasks'
    run
    function to return
    prefect.Task[TRes]
    where TRes is the actually non-wrapped function's return type?
    I'm not sure if Python can do this, but I think
    contextmanager
    annotations does this
    emre

    emre

    1 year ago
    prefect tries its best to infer tasks from builtin python operators. Like
    task1 + task2
    is resolved as
    AddTask(task1, task2)
    by prefect. But resolving arbitrary func calls like
    set(task1)
    to a prefect task is not possible at this point
    haf

    haf

    1 year ago
    On this separate thread of discussion of what to implicitly cast: wouldn't it be more "normal" to provide wrappers for all built-in collections?
    emre

    emre

    1 year ago
    haf

    haf

    1 year ago
    Ok, I must be misunderstanding then. The docs for these say basically "these are implicit and you don't have to use thejm"?
    In general, users will not instantiate these tasks by hand; they will automatically be...
    emre

    emre

    1 year ago
    I believe
    {task1}
    would get resolved into a
    Set(task1)
    task definition, in this sense they are implicit. calling pythons default
    set
    function cannot be detected by prefect however
    haf

    haf

    1 year ago
    So does that mean, had I created a
    def handle(xs: Set[Thing]) -> None:
    I would have gotten the set of things automatically cast?
    emre

    emre

    1 year ago
    The users will not use these tasks by hand bit works like below:
    @task
    def ret1():
        return 1
    @task
    def ret2():
        return 2
    @task
    def print_content(x):
        print(x)
    with Flow("ListTest") as f:
        print_content([ret1, ret2])
    []
    operator created a list of 2 prefect tasks. and prefect, while building the flow, understood that the output of these two tasks should be appended in a list. And inserted a
    List
    task to facilitate this.
    haf

    haf

    1 year ago
    Ok, I see
    Thanks
    emre

    emre

    1 year ago
    a
    list
    call however would try to iterate over tasks, because the
    list
    function assumes its inputs are iterable. You will simply get the above error, even before building a python list of tasks. Because task objects are not iterable.