Hi everyone! My company is thinking of using Prefect but we'd like to talk through some of our use cases and see if it's a good fit. Who's the best person to talk with?

Hi @Adam! Welcome to Prefect!! 🚀 Feel free to share a bit here if you like for community input, but if you are looking more for someone to talk directly to you can email david at prefect dot io to set up an intro call; a technical resource will be on the call too to walk through your engineering specifics as necessary 🙂

Hi @Adam, welcome to the community! I'm CTO at a healthcare startup doing machine learning and data engineering pipelines with Prefect. We've been using Prefect for a while (more than a year) and would be happy to share our experience and answer questions. If it's helpful, feel free to post here or DM me any time.

Thanks @Laura Lorenz (she/her) and @Joe Schmid. After reading through the docs it seems Prefect would be a great fit for our needs. A follow up question though. I'm trying to understand the different options for deploying Prefect. We'd like to try Prefect Cloud + use one of our existing Kubernetes cluster to run the jobs. I'm just a bit confused by the "Execution Environments" and "Agents" section in the Orchestration docs. How do these concepts relate to each other? Do you have a guide to a best practice setup for running on Kubernetes?

Hi @Adam, there are a ton of different ways to deploy Prefect as overall we try to be platform agnostic, but as a result it can be hard to get started on one specific deployment 🙂 Based on what you are saying I would recommend a Kubernetes Agent deployed in yoru cluster, and if you want to customize the worker jobs that spawn from that agent you can do so with the KubernetesJobEnvironment (or the DaskKubernetesEnvironment if you want high parallelism for your tasks by leveraging Dask). Here are a few other resources I recommend: blog on deploying with Prefect cloud + Kubernetes on AWS: https://medium.com/the-prefect-blog/seamless-move-from-local-to-aws-kubernetes-cluster-with-prefect-f263a4573c56 execution layer demo video () and companion guide (https://coda.io/@laura-lorenz/setting-up-your-execution-layer-a-companion-guide) the last slide in these slides from Pycon 2020 open space, which has an architecture diagram of Cloud + your execution layer: https://docs.google.com/presentation/d/1TfOsYmsjgbwXRkiItb2ZeTW_oYxXWAWKMtEnEFOyPiA/edit?usp=sharing

Thanks @Laura Lorenz (she/her). The KubernetesAgent does indeed sound like a great place to start for us. Regarding the jobs it creates, I assume it uses the Docker container that was built right?

Exactly, it will use the Docker container built for your flow storage. (Here’s a github ref if you are interested for where that image starts getting set in the agent’s code: https://github.com/PrefectHQ/prefect/blob/f250e90340480d285fce8d25d5139a943e3d42a4/src/prefect/agent/kubernetes/agent.py#L115). Glad you like the videos! 😇

Thank you @Laura Lorenz (she/her), that makes a lot more sense now!