https://prefect.io logo
Powered by
Title
c

Camilo Fernandez

05/31/2022, 7:59 AM
Hi Everyone. I have deployed a prefect server with a Kubernetes agent using the Helm chart in a EKS cluster. Now I'm kind of confused how to register the flow? Where is the Python code supposed to be in Kubernetes?
a

Anna Geller

05/31/2022, 9:35 AM
Either baked into your container image (using Docker storage) or using another storage mechanism such as GitHub or S3 - this page will explain it more but in short: you point Prefect at a GitHub or S3 file containing your flow - this way, Prefect pulls your code at runtime and never sees your code or data (hybrid execution model)
c

Camilo Fernandez

05/31/2022, 11:33 AM
Thank you @Anna Geller for always answering the many questions. Now I understand the storage of flows. We have decided to use the S3 bucket for it and we have created an image in ECR. But I'm following this Medium post of yours and I'm wondering where is the registration of the flow is done, meaning in which prefect service shell is this command 
prefect register --project jaffle_shop -p flows/
supposed to be run in. Since Prefect doesn't need to store the flow code in its server, then how would the Kubernetes agent have a reference to the code the first time? The link you provided led me to this other page and choose the Script based cloud storage. 
#my-flow.py

with Flow("kubernetes-flow", schedule=schedule, storage=S3(stored_as_script=True, key="kubernetes_flow.py", bucket="prefect-k8s-dev"),\
    run_config=KubernetesRun(image=<image-ulr:tag>)) as flow:

    print('hello-world')
   

flow.register(project_name="jaffle_shop")
So, I made some progress by doing the following: 1. Understanding the registration process. It can be run on my dev environment as long as my prefect client is configure to access the Apollo endpoint. I had if configure for other Prefect server. So I changed it in 
~/.prefect/config.toml
I then checked it was getting the project I created in the UI with 
prefect get projects
2. Create custom job template and passing it to the KubernetesRun Config with the parameter 
job_template_path
and added it to the Helm values with 
agent.jobTemplateFilePath
. I didn't confirmed if that was necessary though. 
KubernetesRun(
    job_template_path="myjob.yaml",
    image="public.ecr.aws/123456/my-image:tag")
3. Register the flow in my dev environment with 
prefect register --path my-flow.py --project my-project -l my-label
and enjoyed the view of the jobs being created.
k

Kevin Kho

05/31/2022, 2:29 PM
Yes that’s right. Run the command on local pointing to server. It will get uploaded to S3 if using S3 storage
#prefect-communityJoin Slack
Powered by