Hi. New to Prefect, first time posting.I have a flow which pulls data from an API. The first call retrieves a list of json, then
that list is mapped for processing. IE
with flow("api") as flow:
client = get_client(api_key, api_secret)
main = get_data(client)
mapped = get_mapped_data.map(prefect.unmapped(client), main)
The issue is that sometimes the flow can run for more than an hour,
at which time the access token for the API expires. (and 1 hour is max
that issuer allows) I can catch the error in a mapped task and try doing a token refresh.
But I'm unsure whether the new token will be available to mapped tasks, or
if each mapped task will end up requesting it's own new token, or something
else different entirely.I guess I'm asking what the best way is to address this problem.Thanks!
10 months ago
Hey @Kendal Burkhart, I think you can maybe use the KV Store to persist that token, and then the tasks can pull the token from the KV Store if it is still valid and use it for the API. Docs for that
Ultimately, you need some way to share this token between multiple threads so it has to be pulled from somewhere I think