Thread
#prefect-server
    ash

    ash

    1 year ago
    Hi everyone, I deployed prefect on a single node and now aiming at deploying the same on kubernetes using helm chart. While learning from live sessions on youtube I came across this piece of code.
    @task
    	def say_hello():
    		print("hello world !!!")
    
    with Flow("hello world") as flow:
    	say_hello()
    
    
    flow.storage = Docker(registry_url = "", image name = "hello world flow")
    flow.register("demo")
    Can someone please explain to me 1.) Why we used container registry for storing a flow and what difference would it make compared to other script based storage like github. 2.) When we execute a flow whose server is present on K8 but no executer is defined like in above snippet, where would the flow be executed on local machine or on a pod on the cluster.
    m

    Mariia Kerimova

    1 year ago
    Hello! If you are using Docker storage, your flow will be stored in Docker image (it will be created for you), then this image will be pushed to the registry (which you will provide). For some users, it's preferable storage option, but you could use other storage types described here. Are you using KubernetesRun?
    Kevin Kho

    Kevin Kho

    1 year ago
    For 1 specifically, the pros and cons of script based storage and pickle based storage are here . Pickle based is needed if you’re packaging up other dependencies along with your code.
    For 2, it will be the pod on the cluster. No executor defaults to LocalExecutor which runs things sequentially.
    ash

    ash

    1 year ago
    @Kevin Kho for 2, so if i am running 10 different flows at once , will 10 different pods be deployed on cluster.
    Kevin Kho

    Kevin Kho

    1 year ago
    Yes that’s right
    ash

    ash

    1 year ago
    Ok but isn't that a bad practice as in some kind wastage of resources. It might happen that a pod is capable of running the 3-4 flows but we are still deploying one for each.
    @Mariia Kerimova actually i have not defined any runconfig in the snippet yet.
    Kevin Kho

    Kevin Kho

    1 year ago
    The
    RunConfig
    will let you determine the resources for that flow like
    flow.run_config = KubernetesRun(cpu_request=2, memory_request="2Gi")
    The KubernetesRun config docs are the in the second link Mariia gave there
    ash

    ash

    1 year ago
    Got it
    RunConfig objects define where and how a flow run should be executed. Each RunConfig type has a corresponding Prefect Agent (i.e. LocalRun pairs with a Local Agent, DockerRun pairs with a Docker Agent, ...)
    so when i use helm charts and i am deploying agent on the same kubernetes cluster , then my runconfig will also be kubernetesRun
    And if i want to run the flow on my local machine then i need to have a local agent , thats what i am understanding from the above mentioned runconfig lines, Please correct me if I am wrong.
    Also is it possible to run flow in local machine despite agent being on cluster pod ?
    Kevin Kho

    Kevin Kho

    1 year ago
    Yes to KubernetesRun when running on the cluster. Also use the Kubernetes Agent. Local agent for local machine is also right. I think helm would be separate from these concepts as it’s for installation
    You would need a separate local agent to pick up that flow for the local machine
    You can have separate labels to distinguish which agents can pick up which flows
    ash

    ash

    1 year ago
    Sorry, but did not understand what you meant by
    helm would be separate from these concepts as it's for installation
    Kevin Kho

    Kevin Kho

    1 year ago
    What are you using helm for?
    ash

    ash

    1 year ago
    I am using helm chart config from prefectHq, to deploy prefect agent along with prefect server on kubernetes
    Kevin Kho

    Kevin Kho

    1 year ago
    Ah ok so I meant that yes that’s right. It’s just a separate concept from the k8s RunConfig. I think you’re good.
    ash

    ash

    1 year ago
    So now all i would need to do is have two agents , one on local and one on kubernetes, and based on their labels i can decide whether the flow runs on local machine or kubernetes pods
    Kevin Kho

    Kevin Kho

    1 year ago
    Yes that’s right. Labels provide the separation
    ash

    ash

    1 year ago
    great , will work on this and also the storage thing. Thanks a lot Kevin and Mariia