Hello Prefect community, just quick question regarding the process where the flow source is run thr...
c
Hello Prefect community, just quick question regarding the process where the flow source is run through after downloading it from source (but before executing the tasks). My set up is as follows: 1. KubernetesAgent 2. KubernetesRun with a custom
job_template.yaml
Is the flow download process: 1. Run on the Kubernetes job spun up by the Agent (and hence takes the environment variables as specified in
job_template.yaml
); OR 2. Run on the agent, which then passes the flow metadata to the spun up job?
g
The agent triggers a Kubernetes Job with the
job_template.yaml
you've defined
c
Thanks Guy. That part is clear to me. What I am less clear about is the "flow source" read process that occurs twice in the Prefect lifecycle 1. Once upon registration; 2. Another time upon flow run. In particular, where does this occur after the Kubernetes Agent receives a scheduled flow?
g
From looking at the agent's logs in my k8s cluster it looks like that the downloading and running of the flow file is done inside the k8s job and not the agent
1
c
Thank you Guy, this is very helpful 🙏
g
@Guy Maliar Hey Guy, I think the "flow source" exec process actually happens here (as part of the storage logic rather than the flow execution logic ) https://github.com/PrefectHQ/prefect/blob/c9ac90956ef8189f3d455299f2501d0735f4fc7b/src/prefect/utilities/storage.py#L88