Hey again everyone, is anyone implementing prefect...
# prefect-community
j
Hey again everyone, is anyone implementing prefect while initializing local agents inside kubernetes pods? When our kubernetes pods die off and restart (for whatever reason) the old agents continue to live in a “dead” state in prefect. Eventually the agents accumlate and slow the whole UI down and deteriorate the performance. Anyone else have this problem or know a way to manage the agents in code?
1
k
A bit curious why the local agent in the Kubernetes pods? I personally haven’t seen anyone do it but wouldn’t know if there is some here who does
j
We initialize them in the start script for the ETL…
Copy code
prefect agent local start -l ${EDW} -l chizzle -a ${PREFECT_SERVER__APOLLO_URL} --show-flow-logs &
We’re looking for a way to use the prefect agents, but we have to find a way to copy the whole DBT code repo to the agents
We end up with a
local-agent
for each replication instance of the ETL and they eventually die out then accumulate.
k
You could clone the dbt repo into the Kubernetes pod through the image entrypoint or through Python code. Have you seen that setup?
🙌 1
j
Is there examples or documentation to look at?
k
The post above is a link to a blog
j
lol you linked to it
Thanks!
I’ll check it out