Hi all!
Does anyone have experience with controlling BigQuery reservations (Flex slots) from Prefect? Any best practices to share? I'm especially wondering how to ensure slots are canceled after a run.
My basic use case would be to lower our BQ costs on our dbt runs:
1. Purchase flex slots
2. Run dbt models
3. Cancel flex slots
Thanks for your input!
a
Anna Geller
02/23/2022, 10:03 AM
This article shows how you can create reservations directly in Python (e.g. from a Prefect flow) and how you can delete those with a cleanup function. Perhaps you could use the cleanup function with a resource manager?
a
Andreas
02/23/2022, 10:06 AM
Thank you @Anna Geller the resource manager sounds promising!
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.