Hi. New to Prefect, first time posting.
I have a flow which pulls data from an API. The first call retrieves a list of json, then
that list is mapped for processing. IE
Copy code
with flow("api") as flow:
client = get_client(api_key, api_secret)
main = get_data(client)
mapped = get_mapped_data.map(prefect.unmapped(client), main)
The issue is that sometimes the flow can run for more than an hour,
at which time the access token for the API expires. (and 1 hour is max
that issuer allows) I can catch the error in a mapped task and try doing a token refresh.
But I'm unsure whether the new token will be available to mapped tasks, or
if each mapped task will end up requesting it's own new token, or something
else different entirely.
I guess I'm asking what the best way is to address this problem.
Thanks!
k
Kevin Kho
11/16/2021, 7:44 PM
Hey @Kendal Burkhart, I think you can maybe use the KV Store to persist that token, and then the tasks can pull the token from the KV Store if it is still valid and use it for the API. Docs for that
Kevin Kho
11/16/2021, 7:48 PM
Ultimately, you need some way to share this token between multiple threads so it has to be pulled from somewhere I think
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.