Hi Everyone. I have deployed a prefect server with...
# prefect-community
c
Hi Everyone. I have deployed a prefect server with a Kubernetes agent using the Helm chart in a EKS cluster. Now I'm kind of confused how to register the flow? Where is the Python code supposed to be in Kubernetes?
a
Either baked into your container image (using Docker storage) or using another storage mechanism such as GitHub or S3 - this page will explain it more but in short: you point Prefect at a GitHub or S3 file containing your flow - this way, Prefect pulls your code at runtime and never sees your code or data (hybrid execution model)
c
Thank you @Anna Geller for always answering the many questions. Now I understand the storage of flows. We have decided to use the S3 bucket for it and we have created an image in ECR. But I'm following this Medium post of yours and I'm wondering where is the registration of the flow is done, meaning in which prefect service shell is this command
prefect register --project jaffle_shop -p flows/
supposed to be run in. Since Prefect doesn't need to store the flow code in its server, then how would the Kubernetes agent have a reference to the code the first time? The link you provided led me to this other page and choose the Script based cloud storage.
Copy code
#my-flow.py

with Flow("kubernetes-flow", schedule=schedule, storage=S3(stored_as_script=True, key="kubernetes_flow.py", bucket="prefect-k8s-dev"),\
    run_config=KubernetesRun(image=<image-ulr:tag>)) as flow:

    print('hello-world')
   

flow.register(project_name="jaffle_shop")
So, I made some progress by doing the following: 1. Understanding the registration process. It can be run on my dev environment as long as my prefect client is configure to access the Apollo endpoint. I had if configure for other Prefect server. So I changed it in
~/.prefect/config.toml
I then checked it was getting the project I created in the UI with
prefect get projects
2. Create custom job template and passing it to the KubernetesRun Config with the parameter
job_template_path
and added it to the Helm values with
agent.jobTemplateFilePath
. I didn't confirmed if that was necessary though.
Copy code
KubernetesRun(
    job_template_path="myjob.yaml",
    image="public.ecr.aws/123456/my-image:tag")
3. Register the flow in my dev environment with
prefect register --path my-flow.py --project my-project -l my-label
and enjoyed the view of the jobs being created.
k
Yes that’s right. Run the command on local pointing to server. It will get uploaded to S3 if using S3 storage