https://prefect.io logo
s

Simo Tumelius

10/28/2020, 12:08 PM
Hi! I have an ETL flow that utilizes mapping to process each file in a separate branch. The transformation is done in multiple steps and in one of the steps I need to be able to access the input arguments of an upstream task. Example: extract_file -> apply_transformation_1 -> apply_transformation_2 -> apply_transformation_3 (here I need to access input arguments of apply_transformation_1) -> load. Does Prefect offer some built-in functionality for implementing this kind of a "task context" or do I need to create a custom class that contains task arguments and its results and is passed between tasks? Thanks in advance!
j

josh

10/28/2020, 1:31 PM
Hi @Simo Tumelius the sort of task context you are describing is something that only exists in the context of a single task’s run. (i.e. a task’s context is not passed to any other tasks) I think your best bet to accomplish what you are after is to take whichever inputs that are passed into
apply_transform_1
and pass them through to
apply_transform_3
. Or in a different sense take the results of
extract_file
(which are the inputs to transform_1) and pass them directly to the transform_3 in the downstream
s

Simo Tumelius

10/28/2020, 1:32 PM
Got it, thanks @josh!