https://prefect.io logo
Title
d

Deceivious

03/06/2023, 9:17 AM
Hello everyone, Just wondering if prefect supports adding additional logs to the flow logs. I am running a cached task with the following parameters.
cache_key_fn=task_input_hash,
    cache_expiration=datetime.timedelta(hours=2),
    persist_result=True,
    result_storage=LocalFileSystem(), # Change to remote in future
    result_serializer=JSONSerializer()
For every task call , I want to log the
cache_key
, if the task returned the cached result OR if the result were cached. Is this supported by prefect by default? / What are the best practices for this type of logging to be implemented?
f

Filip Panovski

03/06/2023, 9:24 AM
I'm not sure what the Prefect way of doing this would be, but perhaps adding a log message inside your actual task would suffice? As I understand it, the task is not actually "run" when a cached result is available, so nothing should be logged. It's not explicit, but it's something at least..
d

Deceivious

03/06/2023, 9:26 AM
Yes that's one of the issue I have. Any logging code I write within the task will not be executed . I could wrap my own function over
task_input_hash
to log the
cache key
but I have no way of knowing if the result was returned from the cache or not.
f

Filip Panovski

03/06/2023, 9:29 AM
Maybe you can have a log at the log levels: https://docs.prefect.io/concepts/logs/#logging-configuration I'd set everything to e.g.
DEBUG
and check if the caching layer logs anything additionally. If so, you could set the log level of only the cache class to
DEBUG
d

Deceivious

03/06/2023, 9:35 AM
Thanks. Ill try changing the log level and see if anything else in the logs have changed. Also will wait for peeps who have had same issue to comment.
c

Christopher Boyd

03/06/2023, 2:18 PM
@Deceivious, you can add
print_logs=True
to your task / flow decorators to add print statements to your log output. Additionally,
2.7.11
allowed the use of custom loggers - https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md#release-2711
d

Deceivious

03/06/2023, 2:20 PM
@Christopher Boyd this doesn't solve the actual issue I have mentioned. Cuz those logs i want are internals of prefects.
c

Christopher Boyd

03/06/2023, 2:25 PM
I believe it can be retrieved from the task context, but if I can’t verify that, it can be re-computed
it can’t be retrieved from task context, the task context is used as input
d

Deceivious

03/06/2023, 2:46 PM
@Christopher Boyd, This function is for creating a cache key. What I am trying to do is find a way to check if the task result returned was cached or actual result for a task that was run from within the flow.
image.png
Here are 2 flows - the one in left builds the cache and the one in second you can see the cached "bubble" denoting that the results were from cache. I can get these info from UI and API but unsure as to how to get those from within the flow run code itself
c

Christopher Boyd

03/06/2023, 2:50 PM
The cache result is stored cloud side, so when you re-run your flow it computes the cache key, then verifies if it matches (or does not) cloud side
d

Deceivious

03/06/2023, 3:01 PM
Yes, I understand that the task
state
is stored on the database [previously
orion
] , The result itself is stored in memory OR defined blocks depending on how that task decorators are configured. But what I want to know is
if the flow context has any information on if the called task actually ran or took the result out of the storage [cache].
For example: in API, the state_name field changes based on the state and if the results were cached. [left pic : Cached , right pic : completed] . I want to get this in the flow code so i can send a warning if we received cached data.
Also thanks for the patient @Christopher Boyd 😄 I know this is a weird use case .
c

Christopher Boyd

03/06/2023, 3:18 PM
I could be mistaken, but isn’t this already the case in the agent logs that the task reports it was in a cached state? Maybe @Zanie might have a better understanding here as I believe they helped write the cacheing and docs
d

Deceivious

03/06/2023, 3:19 PM
16:19:31.273 | INFO    | Task run '_send_request_with_retry-1' - Finished in state Cached(type=COMPLETED)
I do see the cached log
c

Christopher Boyd

03/06/2023, 3:20 PM
I’d like to be more helpful here, I just don’t quite know the answer
1
d

Deceivious

03/06/2023, 3:26 PM
Thanks @Christopher Boyd. I think ive found it. Seems like if i use
return_state
when calling the task, it does seem to have the state name saying it is
Cached
. I am not sure if the intentended usage of this name is for this purpose though.