Hello everyone , I am a little confused on the below mentioned part

import pymongo
import pandas
from reports.config import mongo_config
import sklearn
from prefect import Flow
from prefect.storage import Docker

@task
	def say_hello():
		print("hello world !!!")

with Flow("hello world") as flow:
	say_hello()


flow.storage = Docker(registry_url = , image name = "hello world flow")
flow.register("demo")

In above code I am importing three external libraries i.e pandas, sklearn, pymongo and

mongo_config

which here contains configuration related information for connecting with mongo

When I register a flow, lets say for code above ,

A.) step(1) A docker image containing everything including external libraries, mongo config and flow code will be built and saved to container registry.
	step(2) Its Metadata including a schedule if any , its path to dependencies from container registry etc will be saved on postgres.
	step(3) When the kubernetes agent polls and have to run the above flow, it will create a pod and dependencies will be installed and after task completion pod is terminated.

	Thats my understanding of how things are working, please correct me if am wrong on any of above.
	Now one thing here is what if mongo config changes, whenever we built pipelines for reporting all we want to do is just change config at one place and  changes   are incorporated for every other report but going on with above approach , i might need to re-register every flow to let it engulf the updated config, Thats what i thinking over here, can you suggest someways of how can i change config at one place so that all the flows knows it and i don't have to re-register all my flows.

B.) One way that i think will be able to solve this is when I use github as storage as the code is read from github so the updated config will also be taken into consideration possibly but there is one issue in this approach ,
when the pod is created to run the script how the dependencies will be installed on the pod  since we don't have docker image this time?

Hello! If you're using prefect image, you'll have to specify additional dependencies in your run config. You can find the example here. If you are using Kubernetes, yes, the flow will run in the pod, which by default will be deleted upon termination. I think you can use parameters for mongo_config. You can find info about using Parameters here.

Hey @ash, there is this feature in Cloud called KV store where you can store key value pairs. You could store the database connection there (assuming nothing sensitive). For your use case you since you're on server, you can replicate this by storing it in something like Google Firestore where you just retrieve some key value pairs during runtime when you need it

Hey @Mariia Kerimova, if i use parameter, i will again have to specify defaults(i don't want to store sensitive value in scripts) for the flows that are scheduled and of course writing parameter for each flow would be very tough. Hey @Kevin Kho lets assume the credentials are sensitive since it can be anything let's say prod related credentials.

Prefect Cloud has Secrets where you can store sensitive stuff and then retrieve them in your flow. For server, you can use environment variables and Secrets, otherwise you need to do something more involved like setting up your own secret store

ohh ok , let me give a read to prefect secrets once. This might possibly resolve my first problem.

No it won't work because Github storage right now just downloads the file, not the whole repo so that file won't exist by default. You need to explicitly download it in your code. This might be easier if it's in something like AWS S3 or an equivalent.

Can you please explain how AWS S3 will help in this😅

Actually ignore S3. Really the Google Firestore type of database might be better than S3

Ok one last thing Kevin, forget the credentials problem, If i want to use github , how will i create an environment for my pods, like docker images have both flow and environment but no such thing is there for github.

Yeah then you need DockerStorage for that. You can’t use GithubStorage if you need to supply an image.

So what you mean here is If I want to run any flow on kubernetes with my prefect server on kubernetes, I cannot do it without using DockerStorage.

Oh sorry! I believe you can use Github storage if you supply an image to KubernetesRun

one thing I am confused about is when I run a flow with storage as github and run_config as kubernetesRun, when the flow is actually run, will it connect with my machine to get the image or will it store on container registry?

It will look locally for that image and if it doesn’t exist, it will try to pull from the registry and if it can’t find it, it wont work.

Is there a source where i can read more about

It will look locally for that image and if it doesn't exist, it will try to pull from the registry and if it can't find it, it wont work.

Thanks a lot Kevin🙌😊

So there is this: https://docs.prefect.io/orchestration/flow_config/storage.html#docker

ok great, I will read this again. Thank you.