Jordan Charlier

    Jordan Charlier

    2 months ago
    Hi everyone ! I have a question about Flow of flow. I’m trying to do flow dynamically and sequentially because I don’t have a huge server at my disposal. I’m trying to do what’s on the picture. I don’t know how much sources I’ll have. And the ETL is not parallel. Is it possible with prefect, please ? Thanks a lot
    Anna Geller

    Anna Geller

    2 months ago
    yes, it's possible! if you can do it in Python, you can orchestrate it with Prefect. How would you write this in Python?
    Jordan Charlier

    Jordan Charlier

    2 months ago
    To do it, I would have put the 3 functions (tasks) inside a for loop
    _, list_db = get_list_db()
    for db in list_db:
       e = extract(db)
       t = transform(e)
       l = load(t)
    I have a typing error for list_temp_file : `Task is not iterable. If your task returns multiple results, pass
    nout
    to the task decorator/constructor, or provide a
    Tuple
    return-type annotation to your task.` Even if I decorate get_list_db with nout=2
    Anna Geller

    Anna Geller

    2 months ago
    A for loop won't work in 1.0. try the same in Prefect 2.0 as follows:
    @flow
    def your_flow():
        _, list_db = get_list_db()
        for db in list_db.result():
            e = extract(db)
            t = transform(e)
            l = load(t)
    Jordan Charlier

    Jordan Charlier

    2 months ago
    Thanks Anna !