Hello Prefect community, just quick question regarding the process where the flow source is run through after downloading it from source (but before executing the tasks).
My set up is as follows:
1. KubernetesAgent
2. KubernetesRun with a custom
job_template.yaml
Is the flow download process:
1. Run on the Kubernetes job spun up by the Agent (and hence takes the environment variables as specified in
job_template.yaml
); OR
2. Run on the agent, which then passes the flow metadata to the spun up job?
g
Guy Maliar
09/29/2021, 11:19 AM
The agent triggers a Kubernetes Job with the
job_template.yaml
you've defined
c
Chris L.
09/29/2021, 11:46 AM
Thanks Guy. That part is clear to me. What I am less clear about is the "flow source" read process that occurs twice in the Prefect lifecycle 1. Once upon registration; 2. Another time upon flow run. In particular, where does this occur after the Kubernetes Agent receives a scheduled flow?
g
Guy Maliar
09/29/2021, 11:51 AM
From looking at the agent's logs in my k8s cluster it looks like that the downloading and running of the flow file is done inside the k8s job and not the agent
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.