https://prefect.io logo
Channels
Powered by
m

Matt Drago

12/16/2020, 12:00 AM
Hey folks, I'm looking for some guidance on using the 
GCRResult
. I have Prefect running in a kubernetes cluster, using the 
KubernetesRun
with 
Docker
storage. I've set the created the 
GCRResult
using:
Copy code
result = GCSResult(
    bucket='REDACTED',
    location='flow-results/{flow_name}/{date:%Y/%m/%d}/{task_name}_{task_run_id}.prefect'
)
And assigned the result to the flow with:
Copy code
with Flow("Redacted", schedule=schedule, result=result) as flow:
Weird thing is that the bucket name is being used to store the Result files, but not using the location template that I have provided. One thing that I noticed was that for one task (a function with the 
@task
decorator), I passed in an argument with the name 
date
and that Task did have it's Results stored in the location according to the template.
c

Chris White

12/16/2020, 5:43 AM
Hi @Matt Drago - are you putting different 
results
on your tasks, or are you specifying a 
target
on your tasks? If not, we might need to transition to a GitHub issue with a reproducible example to debug further
m

Matt Drago

12/16/2020, 5:45 AM
I'm only setting a 
result
on the flow (note 
result
singular not plural) and I'm not setting a 
target
on any of the Tasks.
c

Chris White

12/16/2020, 5:53 AM
Hm yea then this sounds like a possible issue; for your own knowledge, this is the line where formatting occurs: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/task_runner.py#L880-L893 As you can see there, the formatting kwargs are taken first from the flow’s parameters, then context, then finally from the task’s own keyword arguments. If you can create a minimal reproducible example and open an issue on GitHub we can get to the bottom of it! Feel free to tag me 
@cicdw
if you do open one
m

Matt Drago

12/16/2020, 5:55 AM
Sweet, thanks for the pointer to the code as well. I'll try and get an example working and then raise an issue.
👍 1
Hey @Chris White, I spent some more time looking at the code that does the formatting and what I was getting. I misinterpreted my results. Prefect is working as intended. The thing that threw me was when I was setting the argument on a task to have the same name as what was in the context. The location template that I provide is being used and the results are going into the right place. On https://docs.prefect.io/core/concepts/results.html#templating-result-locations where it says:
Copy code
alternatively, you can provide a location template format string which will be templated with values in Prefect context at runtime.
Do you think it would be worth changing this to mention that the template will use the values from the run's parameters, the context and then that tasks arguments and explain the preference for which set has precedence?
Thanks for helping me out on this one.
c

Chris White

12/16/2020, 4:24 PM
Yea, I definitely think we should update the wording there; and anytime! Glad you were able to identify the issue — I’ll archive this thread to GitHub for others’ reference
@Marvin archive “Result location template not being templated as I expect”
m

Marvin

12/16/2020, 4:24 PM
7 Views
Powered by