https://prefect.io logo
Join Slack
Channels
Powered by
What is the best way to use a single client object...
# ask-community
t
What is the best way to use a single client object across some mapped tasks?
k
Are you on DaskExecutor or LocalDaskExecutor?
t
LocalDaskExecutor
I currently have the client created INSIDE the mapped tasks, but I hit a new rate limit on the authentication requests and I need to try and auth less.
If I only have a single client object it would be ideal, but I can't think of a good way to do it
k
Try 1. turning off checkpointing for the task that returns the connection 2. using threads for the LocalDaskExecutor and that might work
z
If you’re using threads you might be able to access a globally cached client
t
what is the checkpointing?
k
The Results. 
@task(checkpoint=False)
t
Do I want to map the client object? I would think not.
k
Use 
unmapped
for that guy
t
Oh wow that was easy. Thanks!
p
I had to use different clients across mapped tasks for other reasons and solved the rate limit issue by rate limiting each client to a lower number. For example, if the API accepts 100 calls per minute and i am using concurrency of 5, I'd configure each client instance to rate limit at 20 instead of 100.
7 Views
Open in Slack
PreviousNext
Powered by