Hi, I'm John, a data scientist / software engineer. I generally work in the energy trading space, however I'm currently looking for a job and working on my algorithmic trading system. Successful algorithmic trading requires research, which requires data, which requires pipelines, which require orchestration, and that's what I'm using prefect for. I run prefect on a single node k8s cluster made from a second hand server I bought from ebay, and I store the data on a 4 node ceph cluster. Currently figuring out how to get prefect working a bit more efficiently on my setup, now that I can use genAI to develop data pipelines faster.

hey John, welcome! in case it's helpful, here's my (agent friendly) single node k8s prefect oss setup

Thanks @Nate - that's super useful!