Matthew Webster
01/27/2022, 6:10 PMKevin Kho
Matthew Webster
01/27/2022, 11:47 PMKevin Kho
USER1-KEY
, USER2-KEY
so that if you can just do param+_KEY. There is a thread here I think will be useful one secMatthew Webster
01/28/2022, 3:06 AMKevin Kho
build=False
to not build the Docker storageMatthew Webster
01/28/2022, 4:07 PMKevin Kho
Matthew Webster
01/31/2022, 4:55 AMif __name__ == "__main__":
flow_list = [setup_flow, semester_flow, daily_flow, hourly_flow]
project_list = ["client_1", "client_2"]
for flow in flow_list:
docker_storage.add_flow(flow)
for project_name in project_list:
flow.register(project_name=project_name, build=False)
I’ll need to add a way to parameterize to add more projects without a build in the future. I guess the way to do that is to look at the cloud API?Kevin Kho
Matthew Webster
01/31/2022, 4:01 PMKevin Kho
def build_and_register(my_param):
with Flow():
x = Parameter("x", default=my_param)
flow.register(...)
if __name__ == "__main__":
for param in my_param:
build_and_register(param)
then you can parameterize this using click to take in the values from the command line. You can also pass in some config
file that gets loaded in to create these Flows. What I would personally do then is use DockerRun for dependencies but something like S3 for storage. This means registration gives me a new distinct file for every client with the appropriate defaults. If I have to upgrade the Flow, I would use just change it in one place and re-register everything for it to take effect.python myflow.py -arg1 param1 -arg2 param2
so you can use it with CI/CD if that makes sense:Matthew Webster
02/05/2022, 12:09 AMKevin Kho