s

    Simo Tumelius

    1 year ago
    Hi! I have an ETL flow that utilizes mapping to process each file in a separate branch. The transformation is done in multiple steps and in one of the steps I need to be able to access the input arguments of an upstream task. Example: extract_file -> apply_transformation_1 -> apply_transformation_2 -> apply_transformation_3 (here I need to access input arguments of apply_transformation_1) -> load. Does Prefect offer some built-in functionality for implementing this kind of a "task context" or do I need to create a custom class that contains task arguments and its results and is passed between tasks? Thanks in advance!
    j

    josh

    1 year ago
    Hi @Simo Tumelius the sort of task context you are describing is something that only exists in the context of a single task’s run. (i.e. a task’s context is not passed to any other tasks) I think your best bet to accomplish what you are after is to take whichever inputs that are passed into
    apply_transform_1
    and pass them through to
    apply_transform_3
    . Or in a different sense take the results of
    extract_file
    (which are the inputs to transform_1) and pass them directly to the transform_3 in the downstream
    s

    Simo Tumelius

    1 year ago
    Got it, thanks @josh!