Deceivious
05/08/2023, 1:13 PMdef get_storage_parameters() -> Dict:
return {
"persist_result": True,
"result_storage": where_to_store(),
# where_to_store function returns where to store
# e.g for dev env, we store the file in temporary local fs
# for prod maybe s3 buckets
"result_serializer": JSONSerializer(jsonlib="json"),
}
I originally intended to use it as
@task(**get_storage_parameters())
My deployment environment [github workflows] does not actually need the credentials to the s3 buckets hence the credentials are not available to github.
How would I bind the parameters dynamically only on the run time and not the deployment time? I know that the with_options
could be used but I would have to implement the method on every usage of each and every task / also the code is really dirty with implementing with_options
everywhere.
How is the community handling secrets during deployment?Deceivious
05/08/2023, 1:45 PMdef stored_task():
@task()
def task1():
#actual task
return #stuff
return task1.with_options(**get_storage_parameters())()
Deceivious
05/08/2023, 2:47 PM