y

    Yas Mah

    6 months ago
    Hello 🙂 which possibilities are there to access the result of a task and use it in an other operation with the flow, which is not a task:
    @task
    def get_access_paths(base_path:Path):
        return base_path
    
    with Flow("flow") as flow:
        base_path = Parameter("base_path", default=pathlib.Path(__file__).parent.parent.resolve())
        data_access = get_access_paths(base_path)
        files = [str(Path.joinpath(data_access, x)) for x in data_access.glob('*') if x.is_file()]
    
        input = Parameter("input", default=files)
    Kevin Kho

    Kevin Kho

    6 months ago
    Hi @Yas Mah, 🙂 I think the thinking is the other way around. You need to make the non tasks into tasks to defer their execution to run time. Otherwise they will run while the Flow (DAG) is being constructed.
    y

    Yas Mah

    6 months ago
    Thank you @Kevin Kho, it makes sense!