Alexander Melkoff
01/25/2022, 2:24 PMGITLAB_ACCESS_TOKEN
?Anna Geller
01/25/2022, 2:39 PMprefect agent xxx start --env PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN="your_token"
for KubernetesAgent:
env:
- name: PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN
value: 'your_token'
and then reference it in your storage:
from prefect import Flow
from prefect.storage import GitLab
flow = Flow(
"gitlab-flow",
GitLab(
repo="org/repo", # name of repo
path="flows/my_flow.py", # location of flow file in repo
access_token_secret="GITLAB_ACCESS_TOKEN" # name of personal access token secret
)
)
Alexander Melkoff
01/25/2022, 5:40 PMkubectl -n prefect set env deployment.apps/prefect-agent PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN="XYZ"
checked that agent pod was restarted has new env in the list with other envs
Environment:
PREFECT__CLOUD__API: <http://prefect-apollo.prefect:4200/graphql>
NAMESPACE: prefect
IMAGE_PULL_SECRETS: []
PREFECT__CLOUD__AGENT__LABELS: []
JOB_MEM_REQUEST:
JOB_MEM_LIMIT:
JOB_CPU_REQUEST:
JOB_CPU_LIMIT:
IMAGE_PULL_POLICY:
SERVICE_ACCOUNT_NAME: prefect-serviceaccount
PREFECT__BACKEND: server
PREFECT__CLOUD__AGENT__AGENT_ADDRESS: <http://0.0.0.0:8080>
PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN: XYZ
but flow run still fails with this
Failed to load and execute Flow's environment: ValueError('Local Secret "GITLAB_ACCESS_TOKEN" was not found.')
Anna Geller
01/25/2022, 8:01 PMAlexander Melkoff
01/25/2022, 9:35 PMAnna Geller
01/25/2022, 10:08 PMSam Werbalowsky
01/26/2022, 10:02 PMAlexander Melkoff
02/02/2022, 11:23 AMAWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
values created for that account.
Kubernetes Secret
We use Terraform to manage Kubernetes IaC, so it is used to create the secret
# <http://prefect.tf|prefect.tf>
resource "kubernetes_secret" "prefect-secrets" {
metadata {
name = "prefect-secrets"
namespace = kubernetes_namespace.prefect.metadata.0.name
}
data = {
"gitlab-access-token" = local.secrets.gitlab_access_token
"prefect-aws-access-key-id" = local.secrets.prefect_aws_access_key_id
"prefect-aws-secret-access-key" = local.secrets.prefect_aws_secret_access_key
}
}
Instead of storing values in the code, we use references to AWS Secrets manager.
AWS Secrets Manager
Since it is a bad idea to store secret values in the code, we use AWS Secrets Manager to store values. Terraform retrieves them when configuration is applied to the K8s cluster and creates or updates Kubernetes secret
# <http://secrets.tf|secrets.tf>
data "aws_secretsmanager_secret_version" "secrets" {
secret_id = "k8-secrets"
}
locals {
secrets = jsondecode(data.aws_secretsmanager_secret_version.secrets.secret_string)
}
Custom Job Template
Now we add references to Kubernetes secret in custom job template
# prefect-job-template.yaml
apiVersion: batch/v1
kind: Job
spec:
template:
spec:
containers:
- name: flow
env:
- name: PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN
valueFrom:
secretKeyRef:
name: prefect-secrets
key: gitlab-access-token
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: prefect-secrets
key: prefect-aws-access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: prefect-secrets
key: prefect-aws-secret-access-key
Now we can specify that as job_template
in KubernetesRun
, but it would be more convenient to store it in s3 and use job_template_path
instead. In order to achieve that we also need to provide AWS credentials to Prefect's kubernetes agent. That means we need to add references to Kubernetes secret to Prefect's helm_release resource in Terraform
# <http://prefect.tf|prefect.tf>
...
set {
name = "agent.env[0].name"
value = "AWS_ACCESS_KEY_ID"
}
set {
name = "agent.env[0].valueFrom.secretKeyRef.name"
value = kubernetes_secret.prefect-secrets.metadata.0.name
}
set {
name = "agent.env[0].valueFrom.secretKeyRef.key"
value = "prefect-aws-access-key-id"
}
set {
name = "agent.env[1].name"
value = "AWS_SECRET_ACCESS_KEY"
}
set {
name = "agent.env[1].valueFrom.secretKeyRef.name"
value = kubernetes_secret.prefect-secrets.metadata.0.name
}
set {
name = "agent.env[1].valueFrom.secretKeyRef.key"
value = "prefect-aws-secret-access-key"
}
...
⚠️ Older versions of Prefect's helm chart don't have agent.env
section! Took me some time to notice that.
Flow
Now we can use this simple flow to check if everything works.
# workflow.py
import prefect
from prefect import task, Flow
from prefect.storage import GitLab
from prefect.run_configs import KubernetesRun
from prefect.executors import LocalExecutor
gitlab_storage = GitLab(
repo="gitlab-group-name/prefect-test-flow-repo-name",
ref="main",
path="workflow.py",
access_token_secret="GITLAB_ACCESS_TOKEN",
)
kubernetes_run_config = KubernetesRun(job_template_path="<s3://bucket-name/prefect-job-template.yaml>")
local_executor = LocalExecutor()
@task
def hello_task():
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Hello, cloud!")
with Flow("hello-flow") as flow:
hello_task()
flow.storage = gitlab_storage
flow.run_config = kubernetes_run_config
flow.executor = local_executor
Anna Geller
02/02/2022, 11:34 AMSam Werbalowsky
02/02/2022, 1:47 PMAlexander Melkoff
02/03/2022, 10:55 AMSam Werbalowsky
02/07/2022, 3:24 PM