Hi Everyone, I am prototyping an ETL flow where I ...
# prefect-community
Hi Everyone, I am prototyping an ETL flow where I want to read a bunch of files, do some transformation, and write the results in parallel using a
My flow looks something like this:
Copy code
list_of_files = Parameter()
list_of_frames = extract_fn.map(list_of_files) # This is indeed getting executed in parallel
How can I pass both the result of the previous mapped fn and also the input parameters to my transform fn? I understand that I could probably refactor the transform_fn to avoid using the
but wanted to know if there is any other way of handling this.
Copy code
transform_fn.map(list_of_frames, list_of_files)
As you can tell I am very new to prefect. Thanks a lot!
I think what you're asking for is
: https://docs.prefect.io/core/concepts/mapping.html#unmapped-inputs. This lets you wrap an argument to a
call , and prefect will pass that argument in directly, rather than map over it.
Copy code
transform_fn.map(list_of_frames, unmapped(list_of_files))
or are you asking how to get the filename that corresponds with each frame to your
The later. I am trying to get the filename passed to the transform_fn
transform_fn.map(list_of_frames, list_of_files)
should work then.
that seems to give an error -
some reference tasks failed
. Will debug it
Turns out one of the arguments to the
needed to be
and the rest passed in as is
Thanks for your help