Matthew Webster01/27/2022, 6:10 PM
Matthew Webster01/27/2022, 11:47 PM
so that if you can just do param+_KEY. There is a thread here I think will be useful one sec
Matthew Webster01/28/2022, 3:06 AM
to not build the Docker storage
Matthew Webster01/28/2022, 4:07 PM
Matthew Webster01/31/2022, 4:55 AM
I’ll need to add a way to parameterize to add more projects without a build in the future. I guess the way to do that is to look at the cloud API?
if __name__ == "__main__": flow_list = [setup_flow, semester_flow, daily_flow, hourly_flow] project_list = ["client_1", "client_2"] for flow in flow_list: docker_storage.add_flow(flow) for project_name in project_list: flow.register(project_name=project_name, build=False)
Matthew Webster01/31/2022, 4:01 PM
then you can parameterize this using click to take in the values from the command line. You can also pass in some
def build_and_register(my_param): with Flow(): x = Parameter("x", default=my_param) flow.register(...) if __name__ == "__main__": for param in my_param: build_and_register(param)
file that gets loaded in to create these Flows. What I would personally do then is use DockerRun for dependencies but something like S3 for storage. This means registration gives me a new distinct file for every client with the appropriate defaults. If I have to upgrade the Flow, I would use just change it in one place and re-register everything for it to take effect.
so you can use it with CI/CD if that makes sense:
python myflow.py -arg1 param1 -arg2 param2
Matthew Webster02/05/2022, 12:09 AM