https://prefect.io logo
Title
k

Kyle McEntush

08/22/2020, 4:38 PM
Is it possible to use the imperative API to have child tasks that live in multiple files? Ex: File 1 makes a flow that calls a task in File 2. This works so long as the task in File 2 doesn't call another task within File 2. If I try to do that, I get a
ValueError: Could not infer an active Flow context.
c

Chris White

08/22/2020, 5:34 PM
Hi Kyle - I don’t think this is related to file structure at all; tasks shouldn’t be calling other tasks. The Flow is the container that is responsible for stringing other tasks together.
@task
def task_one():
    pass

@task
def task_two():
    task_one() # will raise an error, always
k

Kyle McEntush

08/22/2020, 6:22 PM
I see. My end goal is to take a list of files, map them, and have tasks for extract, transform, load, etc. In my current setup, I take the list of files and call a process() task that was trying to call subtasks for ETL
What's the best way to rearrange this so that I get independent tasks running/tracked for ETL, while still mapping the files?
c

Chris White

08/22/2020, 6:23 PM
you can keep calling
.map
successively to create longer mapped pipelines, e.g.,
with Flow("my-mapped-flow") as flow:
    task_three.map(task_two.map(task_one))
it sounds like that might be what you’re after
k

Kyle McEntush

08/22/2020, 6:27 PM
I'll give that a shot later. Thanks!
c

Chris White

08/22/2020, 6:27 PM
👍 👍
k

Kyle McEntush

08/22/2020, 6:34 PM
@Chris White and in this nested map, should a task fail somewhere in the middle, will it terminate that branch?
c

Chris White

08/22/2020, 6:35 PM
yea all tasks downstream of the single mapped task will enter a
TriggerFailed
state, but other branches of the map will continue uninterrupted
👍 1