delphi
07/29/2020, 9:35 AMjosh
07/29/2020, 11:21 AMDaniel
07/29/2020, 1:48 PMAdam
07/29/2020, 1:56 PMif __name__ == "__main__":
flow.run()
This allows developers to just do python flow.py
and have it run locally (they use the prefect server start
etc commands to run it locally (i.e. when developing and testing).
We also have a CI pipeline that when they commit their code its sent to our different environments. We have a separate build.py
script that imports the above flow.py
and does something like:
flow.storage = Docker(
registry_url="<http://gcr.io/registry/flows|gcr.io/registry/flows>",
python_dependencies=["pandas", "prefect[google,kubernetes]"],
)
flow.register(project_name="prefect-test-1")
The CI server then calls python build.py
instead, which then registers the flow rather than running it. Hope that makes sensedelphi
07/29/2020, 4:07 PMdelphi
07/29/2020, 4:20 PM