Suchindra
02/16/2021, 9:16 PMLocalDaskExecutor
My flow looks something like this:
list_of_files = Parameter()
list_of_frames = extract_fn.map(list_of_files) # This is indeed getting executed in parallel
How can I pass both the result of the previous mapped fn and also the input parameters to my transform fn? I understand that I could probably refactor the transform_fn to avoid using the list_of_files
but wanted to know if there is any other way of handling this.
transform_fn.map(list_of_frames, list_of_files)
As you can tell I am very new to prefect. Thanks a lot!Jim Crist-Harif
02/16/2021, 9:18 PMprefect.unmapped
: https://docs.prefect.io/core/concepts/mapping.html#unmapped-inputs. This lets you wrap an argument to a .map
call , and prefect will pass that argument in directly, rather than map over it.transform_fn.map(list_of_frames, unmapped(list_of_files))
transform_fn
?Suchindra
02/16/2021, 9:20 PMJim Crist-Harif
02/16/2021, 9:22 PMtransform_fn.map(list_of_frames, list_of_files)
should work then.Suchindra
02/16/2021, 9:28 PMsome reference tasks failed
. Will debug ittransform_fn
needed to be unmapped
and the rest passed in as is