https://prefect.io
Join Slack
Hello everyone, Just wondering if prefect supports adding additional logs to the flow logs. I am run...
# ask-community
d
Hello everyone, Just wondering if prefect supports adding additional logs to the flow logs. I am running a cached task with the following parameters.
Copy code
cache_key_fn=task_input_hash,
    cache_expiration=datetime.timedelta(hours=2),
    persist_result=True,
    result_storage=LocalFileSystem(), # Change to remote in future
    result_serializer=JSONSerializer()
For every task call , I want to log the 
cache_key
, if the task returned the cached result OR if the result were cached. Is this supported by prefect by default? / What are the best practices for this type of logging to be implemented?
f
I'm not sure what the Prefect way of doing this would be, but perhaps adding a log message inside your actual task would suffice? As I understand it, the task is not actually "run" when a cached result is available, so nothing should be logged. It's not explicit, but it's something at least..
d
Yes that's one of the issue I have. Any logging code I write within the task will not be executed . I could wrap my own function over 
task_input_hash
to log the 
cache key
but I have no way of knowing if the result was returned from the cache or not.
f
Maybe you can have a log at the log levels: https://docs.prefect.io/concepts/logs/#logging-configuration I'd set everything to e.g. 
DEBUG
and check if the caching layer logs anything additionally. If so, you could set the log level of only the cache class to 
DEBUG
d
Thanks. Ill try changing the log level and see if anything else in the logs have changed. Also will wait for peeps who have had same issue to comment.
c
@Deceivious, you can add 
print_logs=True
to your task / flow decorators to add print statements to your log output. Additionally, 
2.7.11
allowed the use of custom loggers - https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md#release-2711
d
@Christopher Boyd this doesn't solve the actual issue I have mentioned. Cuz those logs i want are internals of prefects.
c
I believe it can be retrieved from the task context, but if I can’t verify that, it can be re-computed
it can’t be retrieved from task context, the task context is used as input
d
@Christopher Boyd, This function is for creating a cache key. What I am trying to do is find a way to check if the task result returned was cached or actual result for a task that was run from within the flow.
Here are 2 flows - the one in left builds the cache and the one in second you can see the cached "bubble" denoting that the results were from cache. I can get these info from UI and API but unsure as to how to get those from within the flow run code itself
c
The cache result is stored cloud side, so when you re-run your flow it computes the cache key, then verifies if it matches (or does not) cloud side
d
Yes, I understand that the task 
state
is stored on the database [previously 
orion
] , The result itself is stored in memory OR defined blocks depending on how that task decorators are configured. But what I want to know is
Copy code
if the flow context has any information on if the called task actually ran or took the result out of the storage [cache].
For example: in API, the state_name field changes based on the state and if the results were cached. [left pic : Cached , right pic : completed] . I want to get this in the flow code so i can send a warning if we received cached data.
Also thanks for the patient @Christopher Boyd 😄 I know this is a weird use case .
c
I could be mistaken, but isn’t this already the case in the agent logs that the task reports it was in a cached state? Maybe @Zanie might have a better understanding here as I believe they helped write the cacheing and docs
d
Copy code
16:19:31.273 | INFO    | Task run '_send_request_with_retry-1' - Finished in state Cached(type=COMPLETED)
I do see the cached log
c
I’d like to be more helpful here, I just don’t quite know the answer
✅ 1
d
Thanks @Christopher Boyd. I think ive found it. Seems like if i use 
return_state
when calling the task, it does seem to have the state name saying it is 
Cached
. I am not sure if the intentended usage of this name is for this purpose though.
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by