Hi Guys, We're looking to use Webhook Storage to store the flows in our local environment. The Prefect server will be hosted in a K8 Cluster, any recommendations for a storage service that would fit that requirement?
k
Kevin Kho
09/01/2021, 11:40 PM
Hey @Omar Sultan, I haven’t really seen people use Webhook storage that much unless they already have their own file storage system. The example in the docs shows Dropbox but let’s see what the community has to say
o
Omar Sultan
09/01/2021, 11:47 PM
HI Kevin, the challenge I am facing is that I want to implement this in a closed environment that is not connected to the internet. We did some experiments between Script and Pickle storage and decided that Pickle storage is better.
So I am now exploring options that would work for us with this setup
Omar Sultan
09/01/2021, 11:48 PM
Hopefully someone from the community will have some ideas 😄
k
Kevin Kho
09/02/2021, 12:49 AM
But why would you use the webhook then? Doesn’t Local storage make the most sense?
o
Omar Sultan
09/02/2021, 9:50 AM
If I understand correctly local storage won't work if I am registering the Flow from a remote client.
Currently the setup is as follows:
• Prefect Running on Kubernetes using Helm
• Flows run using Kubernetes Agent
• Data Engineers can build their flows and register them remotely from their laptops to be scheduled and run on the server.
So local storage would end up storing it on the engineer's laptop
•
k
Kevin Kho
09/02/2021, 1:53 PM
You might be able to use local storage, which would make the path relative to the Docker container