Hi Everyone, I am using Webhook Storage to register flows on our onprem prefect server, and the flows are configured to used Kubernetes Run .. .I am trying to understand who does the actual GET API Call to the webhook to download the file (Agent Pod, Job Pod) and when does that happen? Does it happen when the flow is about to run or when the flow is being registered?
k
Kevin Kho
09/06/2021, 4:47 PM
Hey @Omar Sultan, the agent does the call to storage to download the file. This happens on flow run. Registration just keeps track of the metadata and saves it in the database (where does the flow live). Upon flow run, the metadata is retrieved, including the storage location, and then it pulls it from there.
After it pulls it, it runs it on Kubernetes.
o
Omar Sultan
09/06/2021, 7:07 PM
Hi Kevin, thanks for the clarification. Our initial impression was that the pull was being done by the POD that was created for the Job running the flow. So thanks for that clarification.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.