ash
06/18/2021, 12:08 PMimport pymongo
import pandas
from reports.config import mongo_config
import sklearn
from prefect import Flow
from prefect.storage import Docker
@task
def say_hello():
print("hello world !!!")
with Flow("hello world") as flow:
say_hello()
flow.storage = Docker(registry_url = , image name = "hello world flow")
flow.register("demo")
In above code I am importing three external libraries i.e pandas, sklearn, pymongo and mongo_config
which here contains configuration related information for connecting with mongo
When I register a flow, lets say for code above ,
A.) step(1) A docker image containing everything including external libraries, mongo config and flow code will be built and saved to container registry.
step(2) Its Metadata including a schedule if any , its path to dependencies from container registry etc will be saved on postgres.
step(3) When the kubernetes agent polls and have to run the above flow, it will create a pod and dependencies will be installed and after task completion pod is terminated.
Thats my understanding of how things are working, please correct me if am wrong on any of above.
Now one thing here is what if mongo config changes, whenever we built pipelines for reporting all we want to do is just change config at one place and changes are incorporated for every other report but going on with above approach , i might need to re-register every flow to let it engulf the updated config, Thats what i thinking over here, can you suggest someways of how can i change config at one place so that all the flows knows it and i don't have to re-register all my flows.
B.) One way that i think will be able to solve this is when I use github as storage as the code is read from github so the updated config will also be taken into consideration possibly but there is one issue in this approach ,
when the pod is created to run the script how the dependencies will be installed on the pod since we don't have docker image this time?
Mariia Kerimova
06/18/2021, 1:31 PMKevin Kho
ash
06/18/2021, 1:48 PMKevin Kho
ash
06/18/2021, 1:56 PMKevin Kho
ash
06/18/2021, 2:03 PMKevin Kho
ash
06/18/2021, 2:26 PMKevin Kho
ash
06/18/2021, 2:33 PMKevin Kho
flow.run_config = KubernetesRun(image="example/image-name:with-tag")
ash
06/18/2021, 2:38 PMKevin Kho
ash
06/18/2021, 2:43 PMIt will look locally for that image and if it doesn't exist, it will try to pull from the registry and if it can't find it, it wont work.
Thanks a lot Kevin🙌😊Kevin Kho
ash
06/18/2021, 3:10 PM