https://prefect.io logo
m

Mikaël Ferreira de Almeida

08/14/2023, 1:11 PM
Hello everyone, I'am quite new with k8s and I don't understand very how it works with Prefect. I have a k8s cluster and I deployed Prefect server, agent and workers using Helm charts.. My target is to deploy my flows and trigger flow runs using the API. I have all my flows code and it's working locally with prefect server. Now I want to deploy my flows in k8s, how I'am I supposed to do it ? The documentation dont lookss clear to me, is there any "full" code example that I can read 🤔 ?
d

Deceivious

08/14/2023, 1:26 PM
look into prefect profile. You need to setup the profile so that the flow code / deployment code uses the API that you have deployed on urs k8s.
h

Henning Holgersen

08/14/2023, 1:49 PM
There are some resources in https://github.com/PrefectHQ/prefect-recipes. My own contribution there (I'm partial) is my kubernetes deployment repo: https://github.com/radbrt/orion_flows/tree/main. I'm working on Azure, so a lot of my readme is about setting up the azure resources specifically. But there is also plenty of k8s stuff.
m

Mikaël Ferreira de Almeida

08/14/2023, 1:55 PM
Thanks both of you ! I didn't understood how simple it was to deploy on any kind of server by updating the
PREFECT_API_URL
variable. Which I find super cool. Now I have to go further on my infrastructure and make it work with all my services and protect the url. I will look on the 2 repos for that
h

Henning Holgersen

08/14/2023, 1:59 PM
Ah, you use your own server! Somehow I missed that. One thing you might be aware of is that the concept of a worker and the
prefect.yaml
file is fairly new, a lot of the repos you see only uses the prefect agent - so far I am not using workers or a prefect.yaml, and I'm happy with it.
d

Deceivious

08/14/2023, 2:14 PM
I personally do not use any of the new features. 100% of my code is python [if not dockerfile]. All deployments is managed by python scripts instead of cli commands / yaml file. But thats just personal preference.
m

Mikaël Ferreira de Almeida

08/14/2023, 2:15 PM
I think I will do everything in python too, it's seems easier to manage my flows
@Henning Holgersen In your k8s cluster your flow code is accessible because you have it inside Docker images that are used by k8s Pods ?
h

Henning Holgersen

08/16/2023, 2:48 PM
@Mikaël Ferreira de Almeida I use azure storage for my flows, you can see in the deployment files I have an azure storage block that I load. In prefect 1 I started out using docker storage, but rebuilding images for every iteration got to be a hassle, and I ended up with very many images. In prefect 2 docker storage works differently, and I am still happy with azure storage.
1
m

Mikaël Ferreira de Almeida

08/16/2023, 2:51 PM
Have you tried GitHub storage ?
h

Henning Holgersen

08/16/2023, 2:54 PM
@Mikaël Ferreira de Almeida not really. At work we have private GitHub repos and machine auth on GitHub is a pain (and I’m happy for every firewall opening I don’t need) so I haven’t gone down that path.
d

Deceivious

08/16/2023, 2:57 PM
We use GitHub as code source but we build images into Azure image registry. And the machine where agents are running can access image in Azure image registry.
GitHub runners build images into the image registry as workflow. And binds the deployment to the new image
You can specify the reference to secrets for access in worker job specs and the encrypted values are stored as secrets in kubernetes
m

Mikaël Ferreira de Almeida

08/16/2023, 3:05 PM
Using an image registry will probably be my final solution. But currently, I wanted to try GitHub or S3 and I'm just having issues saying
FileNotFoundError
so I have to investigate more on that. And I thought it might be due to the fact that I don't use docker images.