Tim Enders

    Tim Enders

    8 months ago
    What is the best way to use a single client object across some mapped tasks?
    Kevin Kho

    Kevin Kho

    8 months ago
    Are you on DaskExecutor or LocalDaskExecutor?
    Tim Enders

    Tim Enders

    8 months ago
    LocalDaskExecutor
    I currently have the client created INSIDE the mapped tasks, but I hit a new rate limit on the authentication requests and I need to try and auth less.
    If I only have a single client object it would be ideal, but I can't think of a good way to do it
    Kevin Kho

    Kevin Kho

    8 months ago
    Try1. turning off checkpointing for the task that returns the connection 2. using threads for the LocalDaskExecutor and that might work
    Michael Adkins

    Michael Adkins

    8 months ago
    If you’re using threads you might be able to access a globally cached client
    Tim Enders

    Tim Enders

    8 months ago
    what is the checkpointing?
    Kevin Kho

    Kevin Kho

    8 months ago
    The Results.
    @task(checkpoint=False)
    Tim Enders

    Tim Enders

    8 months ago
    Do I want to map the client object? I would think not.
    Kevin Kho

    Kevin Kho

    8 months ago
    Use
    unmapped
    for that guy
    Tim Enders

    Tim Enders

    8 months ago
    Oh wow that was easy. Thanks!
    Pedro Machado

    Pedro Machado

    7 months ago
    I had to use different clients across mapped tasks for other reasons and solved the rate limit issue by rate limiting each client to a lower number. For example, if the API accepts 100 calls per minute and i am using concurrency of 5, I'd configure each client instance to rate limit at 20 instead of 100.