Is there a way to specify S3 storage option via config file (and avoid having to specify the bucket explicitly in each flow)?
c
Chris White
12/09/2020, 1:42 AM
Not at this time - we did originally considered config options such as this, but we realized that it quickly led to an explosion of configuration because of the various storage options and their kwargs; you might consider creating a custom thin wrapper class that pulls the bucket from an env var for your own use case
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.