Hi, I'm trying to figure out if I can use Prefect ...
# introductions
j
Hi, I'm trying to figure out if I can use Prefect for an ETL-type scenario. Basically, we'll be receiving large text files via SFTP/HTTPS and we need to process these files eventually into a data warehouse. The Prefect documentation is a bit confusing to me as it mostly focuses on Cloud. I am primarily interested in an on-premise deployment. Would appreciate if anyone can point me to resources that could help me answer these questions: 1. Is Prefect Core = Server + UI? 2. Is there a deployment guide for on-premise that doesn't require Docker/Compose? Thanks!
k
Hey @Joseph Harrison, welcome to Prefect. Prefect Core is just the framework to define flows. Server or Cloud are the backends. The UI is part of Server or Cloud
j
Thanks, Kevin!
k
If you can’t use
docker-compose
, the helm chart instructions are in the README here
j
So basically, the server has to be containerized when deployed?
k
Pretty much because it’s 6 services that interact with each other and need to be on sync version-wise.
Of course, you probably have a reason you can’t use Cloud, but just mentioning Cloud has 10000 free task runs every month for free, which is more than enough to get started with. Is there a reason you can’t use containers?
d
Welcome, @Joseph Harrison!
👍 1
j
@Kevin Kho it's not so much that I can't use containers. My use case is pretty simple and containers add a lot of complexity to the environment. A bit of overkill in my mind. Your point about the 6 services needing to remain in sync is well taken and makes sense. I'll test it out and see if it actually complicates the environment or not. Thanks!