https://prefect.io logo
t

Tim Enders

01/24/2022, 7:10 PM
What is the best way to use a single client object across some mapped tasks?
k

Kevin Kho

01/24/2022, 7:10 PM
Are you on DaskExecutor or LocalDaskExecutor?
t

Tim Enders

01/24/2022, 7:11 PM
LocalDaskExecutor
I currently have the client created INSIDE the mapped tasks, but I hit a new rate limit on the authentication requests and I need to try and auth less.
If I only have a single client object it would be ideal, but I can't think of a good way to do it
k

Kevin Kho

01/24/2022, 7:14 PM
Try 1. turning off checkpointing for the task that returns the connection 2. using threads for the LocalDaskExecutor and that might work
z

Zanie

01/24/2022, 7:14 PM
If you’re using threads you might be able to access a globally cached client
t

Tim Enders

01/24/2022, 7:15 PM
what is the checkpointing?
k

Kevin Kho

01/24/2022, 7:16 PM
The Results.
@task(checkpoint=False)
t

Tim Enders

01/24/2022, 7:18 PM
Do I want to map the client object? I would think not.
k

Kevin Kho

01/24/2022, 7:19 PM
Use
unmapped
for that guy
t

Tim Enders

01/24/2022, 7:20 PM
Oh wow that was easy. Thanks!
p

Pedro Machado

01/25/2022, 12:22 AM
I had to use different clients across mapped tasks for other reasons and solved the rate limit issue by rate limiting each client to a lower number. For example, if the API accepts 100 calls per minute and i am using concurrency of 5, I'd configure each client instance to rate limit at 20 instead of 100.
4 Views