https://prefect.io logo
Title
d

davzucky

09/17/2021, 12:56 AM
Just did a presentation in my company about prefect. All the slide (as Jupyter notebook) and code is available on my github. Please enjoy and feel free to use it if you have a presentation to do. https://github.com/davzucky/prefect_presentation
:upvote: 12
❤️ 9
k

Kevin Kho

09/17/2021, 12:57 AM
@Lucas Hosoya, this might interest your team?
l

Lucas Hosoya

09/17/2021, 1:16 PM
@Kevin Kho Hey! Yup, it does. But we also want some topics other than Dask environments, if possible. Maybe some kubernetes with databricks/spark/hdinsight/glue (run outside of the Agent, so we can put the work on some clusters other than the Agent) But this presentation is great!
d

davzucky

09/17/2021, 2:04 PM
@Lucas Hosoya what are you looking for? Presentation or usage using kubernetes/spark?
l

Lucas Hosoya

09/17/2021, 2:42 PM
@davzucky Hey! Kubernetes/spark would be great to see! Also looking forward to see the usage with a Cloud environment, if possible. Great presentation though, thanks for sharing! PS: I'm actually doing this case as well, but I'd love to see it from someone else!
d

davzucky

09/17/2021, 2:53 PM
We are using all our setup on kubernetes and will run soon spark from prefectv in kubernetes. We need some updated RBAC to get the kuberbetesjobrun task working. Is this what you are looking for?
However I think a presentation about that would be too advance and specialized
1
l

Lucas Hosoya

09/17/2021, 3:00 PM
True! Right now we're about to run kubernetes with databricks instead, due to the spark environment configuration. So everything would run through databricks-connect API, and use the kubernetes agents to orchestrate ETLs/pipelines