Thread
#prefect-community
    Kendal Burkhart

    Kendal Burkhart

    10 months ago
    Hi. New to Prefect, first time posting. I have a flow which pulls data from an API. The first call retrieves a list of json, then that list is mapped for processing. IE
    with flow("api") as flow:
       client = get_client(api_key, api_secret)
       main = get_data(client)
       mapped = get_mapped_data.map(prefect.unmapped(client), main)
    The issue is that sometimes the flow can run for more than an hour, at which time the access token for the API expires. (and 1 hour is max that issuer allows) I can catch the error in a mapped task and try doing a token refresh. But I'm unsure whether the new token will be available to mapped tasks, or if each mapped task will end up requesting it's own new token, or something else different entirely. I guess I'm asking what the best way is to address this problem. Thanks!
    Kevin Kho

    Kevin Kho

    10 months ago
    Hey @Kendal Burkhart, I think you can maybe use the KV Store to persist that token, and then the tasks can pull the token from the KV Store if it is still valid and use it for the API. Docs for that
    Ultimately, you need some way to share this token between multiple threads so it has to be pulled from somewhere I think