https://prefect.io logo
Title
a

Adam

07/23/2020, 4:50 PM
Hi everyone! My company is thinking of using Prefect but we'd like to talk through some of our use cases and see if it's a good fit. Who's the best person to talk with?
👀 1
l

Laura Lorenz (she/her)

07/23/2020, 5:00 PM
Hi @Adam! Welcome to Prefect!! 🚀 Feel free to share a bit here if you like for community input, but if you are looking more for someone to talk directly to you can email david at prefect dot io to set up an intro call; a technical resource will be on the call too to walk through your engineering specifics as necessary 🙂
j

Joe Schmid

07/23/2020, 5:07 PM
Hi @Adam, welcome to the community! I'm CTO at a healthcare startup doing machine learning and data engineering pipelines with Prefect. We've been using Prefect for a while (more than a year) and would be happy to share our experience and answer questions. If it's helpful, feel free to post here or DM me any time.
:marvin: 2
a

Adam

07/27/2020, 9:08 AM
Thanks @Laura Lorenz (she/her) and @Joe Schmid. After reading through the docs it seems Prefect would be a great fit for our needs. A follow up question though. I'm trying to understand the different options for deploying Prefect. We'd like to try Prefect Cloud + use one of our existing Kubernetes cluster to run the jobs. I'm just a bit confused by the "Execution Environments" and "Agents" section in the Orchestration docs. How do these concepts relate to each other? Do you have a guide to a best practice setup for running on Kubernetes?
l

Laura Lorenz (she/her)

07/27/2020, 2:58 PM
Hi @Adam, there are a ton of different ways to deploy Prefect as overall we try to be platform agnostic, but as a result it can be hard to get started on one specific deployment 🙂 Based on what you are saying I would recommend a Kubernetes Agent deployed in yoru cluster, and if you want to customize the worker jobs that spawn from that agent you can do so with the KubernetesJobEnvironment (or the DaskKubernetesEnvironment if you want high parallelism for your tasks by leveraging Dask). Here are a few other resources I recommend: blog on deploying with Prefect cloud + Kubernetes on AWS: https://medium.com/the-prefect-blog/seamless-move-from-local-to-aws-kubernetes-cluster-with-prefect-f263a4573c56 execution layer demo video (

https://youtu.be/50S4RqeEVVo

) and companion guide (https://coda.io/@laura-lorenz/setting-up-your-execution-layer-a-companion-guide) the last slide in these slides from Pycon 2020 open space, which has an architecture diagram of Cloud + your execution layer: https://docs.google.com/presentation/d/1TfOsYmsjgbwXRkiItb2ZeTW_oYxXWAWKMtEnEFOyPiA/edit?usp=sharing
a

Adam

07/27/2020, 3:21 PM
Thanks @Laura Lorenz (she/her). The KubernetesAgent does indeed sound like a great place to start for us. Regarding the jobs it creates, I assume it uses the Docker container that was built right?
And thanks for those videos, love your execution layer demo video!
l

Laura Lorenz (she/her)

07/27/2020, 10:59 PM
Exactly, it will use the Docker container built for your flow storage. (Here’s a github ref if you are interested for where that image starts getting set in the agent’s code: https://github.com/PrefectHQ/prefect/blob/f250e90340480d285fce8d25d5139a943e3d42a4/src/prefect/agent/kubernetes/agent.py#L115). Glad you like the videos! 😇
a

Adam

07/28/2020, 1:34 PM
Thank you @Laura Lorenz (she/her), that makes a lot more sense now!