hey hey, I'm a bit stumped trying to implement a ...
# prefect-communityb
Bob Colner
07/02/2020, 9:58 PMhey hey, I'm a bit stumped trying to implement a data ingestion pattern in prefect.
Is it possible to access task parameter values when templating the result 'target' filename?
I'm trying to have my results stored in a directory structure (aka 'hive' dir partitioning). Thanks for any insight
đź‘€ 1
Bob Colner
07/02/2020, 10:00 PMthe target path should reflect the task parameters if that makes sense
d
Dylan
07/02/2020, 10:06 PM
Hey @Bob Colner!
That definitely makes sense. I don’t believe it’s possible at this time, but let me double-check with the rest of the team before I say that definitively
b
Bob Colner
07/02/2020, 10:07 PMcool
d
Dylan
07/02/2020, 10:08 PM
@Bob Colner Great news! You totally can!
Dylan
07/02/2020, 10:08 PM
The task kwargs are available just like the items in context
b
Bob Colner
07/02/2020, 10:09 PMoh nice, is there an example of how?
d
Dylan
07/02/2020, 10:09 PM
Something like:
Copy code
@task(target="{param}/file.txt")
def my_task(param):
passđź‘Ť 1
Dylan
07/02/2020, 10:09 PM
There is not
Dylan
07/02/2020, 10:09 PM
I’m going to open a ticket to better document this now
Dylan
07/02/2020, 10:11 PM
@Marvin open “Better document the fact that task kwargs can be used to template result targets”
m
Marvin
07/02/2020, 10:11 PMb
Bob Colner
07/02/2020, 10:11 PMI'll give this a try.
d
Dylan
07/02/2020, 10:12 PM
đź‘Ť let us know how it goes!
b
Bob Colner
07/02/2020, 10:12 PMFYI I think this would make a good additional idiom -especially when we can do custom serializers (parquet etc.) common use case (API calls -> dir partitioned csv/parquet
d
Dylan
07/02/2020, 10:13 PM
That’s a great suggestion
b
Bob Colner
07/02/2020, 10:15 PMspeaking of, do you have an idea of when the custom/passthrough serialization will be ready?
d
Dylan
07/02/2020, 10:40 PM
I’m not sure
Dylan
07/02/2020, 10:40 PM
Let me see if I can find the issue
Dylan
07/02/2020, 10:41 PM
Dylan
07/02/2020, 10:42 PM
@Bob Colner looks like it’s on 0.12.1, so you’re good to go!
b
Bob Colner
07/02/2020, 10:44 PMvery cool
Bob Colner
07/02/2020, 10:44 PMso this example in the docs in would work as a 'passthrough'?
Copy code
class MySerializer(prefect.engine.serializers.Serializer):
def serialize(self, value: Any) -> bytes:
# transform a Python object into bytes
pass
def deserialize(self, value: Union[bytes, str]) -> Any:
# recover a Python object from bytes
pass