mot s09/22/2020, 8:01 PM
Jim Crist-Harif09/22/2020, 8:18 PM
would prefect agent be running on the edge (hadoop gateway servers) and prefect backend server on K8s clusterYou could run the backend and agents wherever it makes sense for you to do so. It sounds like you want your flows to run on the edge node, so you'd want to have the agent run there as well. If you're running prefect server, you just need to have it running somewhere where both the agent and the running flows can access its api.
How would flow orchestration workI suggest walking through the tutorial (https://docs.prefect.io/orchestration/tutorial/configure.html) and if you have further questions asking something more specific.
Would agent be kerberos awarePrefect server has no authentication model, so there's no auth to deal with here. If you're running using prefect cloud, we use our own authentication/authorization model, and don't plugin to kerberos. That said, there are definitely ways to have a prefect flow run with valid hadoop delegation token so you can kick off a yarn job on your cluster. And a yarn-backed agent is within scope for prefect (we just haven't written one yet).
mot s09/23/2020, 6:39 PM