https://prefect.io logo
Title
j

Joseph Harrison

09/14/2021, 12:11 AM
Hi, I'm trying to figure out if I can use Prefect for an ETL-type scenario. Basically, we'll be receiving large text files via SFTP/HTTPS and we need to process these files eventually into a data warehouse. The Prefect documentation is a bit confusing to me as it mostly focuses on Cloud. I am primarily interested in an on-premise deployment. Would appreciate if anyone can point me to resources that could help me answer these questions: 1. Is Prefect Core = Server + UI? 2. Is there a deployment guide for on-premise that doesn't require Docker/Compose? Thanks!
k

Kevin Kho

09/14/2021, 12:13 AM
Hey @Joseph Harrison, welcome to Prefect. Prefect Core is just the framework to define flows. Server or Cloud are the backends. The UI is part of Server or Cloud
j

Joseph Harrison

09/14/2021, 12:14 AM
Thanks, Kevin!
k

Kevin Kho

09/14/2021, 12:14 AM
If you can’t use
docker-compose
, the helm chart instructions are in the README here
j

Joseph Harrison

09/14/2021, 12:17 AM
So basically, the server has to be containerized when deployed?
k

Kevin Kho

09/14/2021, 12:24 AM
Pretty much because it’s 6 services that interact with each other and need to be on sync version-wise.
Of course, you probably have a reason you can’t use Cloud, but just mentioning Cloud has 10000 free task runs every month for free, which is more than enough to get started with. Is there a reason you can’t use containers?
d

Dylan

09/14/2021, 1:26 PM
Welcome, @Joseph Harrison!
👍 1
j

Joseph Harrison

09/14/2021, 1:31 PM
@Kevin Kho it's not so much that I can't use containers. My use case is pretty simple and containers add a lot of complexity to the environment. A bit of overkill in my mind. Your point about the 6 services needing to remain in sync is well taken and makes sense. I'll test it out and see if it actually complicates the environment or not. Thanks!