2j
10/06/2021, 2:27 AMAZURE_STORAGE_CONNECTION_STRING
for access to blob storage. I'm not seeing anything baked into the helm values.yaml (though maybe something in prefectConfig
could work?). I was thinking of attaching a k8s secret after the helm deployment? But I think it needs to be set on the job that the agent kicks off. And I should be able to set the env var as-is, or in secret form like PREFECT__CONTEXT__SECRETS__AZURE_STORAGE_CONNECTION_STRING
?-e
on https://github.com/PrefectHQ/server/blob/master/helm/prefect-server/templates/agent/deployment.yaml#L50Thomas Fredriksen
10/06/2021, 5:57 AMwith Flow(...) as flow
for example).job_config.yaml
that is with the agent. The Helm-chart does not have this built in, so you would need to disable the agent in the helm-chart and create your own kubernetes deployment with a config-map holding the job_config.yaml
.job_template.yaml
here is quite sparse, but the default template in the Helm-chart is nice, however the empty volume mounts caused an issue for us. Instead, here is a template that will work:
apiVersion: batch/v1
kind: Job
metadata:
name: prefect-job
labels: {}
spec:
template:
metadata:
labels: {}
spec:
containers:
- name: flow
image: prefecthq/prefect:latest
imagePullPolicy: IfNotPresent
args:
- prefect
- execute
- flow-run
env:
- name: PREFECT__CLOUD__API
value: PREFECT__CLOUD__API
- name: PREFECT__CLOUD__AUTH_TOKEN
value: PREFECT__CLOUD__AUTH_TOKEN
- name: PREFECT__CONTEXT__FLOW_RUN_ID
value: PREFECT__CONTEXT__FLOW_RUN_ID
- name: PREFECT__CONTEXT__FLOW_ID
value: PREFECT__CONTEXT__FLOW_ID
- name: PREFECT__CONTEXT__NAMESPACE
value: PREFECT__CONTEXT__NAMESPACE
- name: PREFECT__CLOUD__AGENT__LABELS
value: PREFECT__CLOUD__AGENT__LABELS
- name: PREFECT__CLOUD__SEND_FLOW_RUN_LOGS
value: PREFECT__CLOUD__SEND_FLOW_RUN_LOGS
# Backwards compatibility variable for `SEND_FLOW_RUN_LOGS`
- name: PREFECT__LOGGING__LOG_TO_CLOUD
value: PREFECT__LOGGING__LOG_TO_CLOUD
- name: PREFECT__LOGGING__LEVEL
value: "INFO"
- name: PREFECT__CLOUD__USE_LOCAL_SECRETS
value: "false"
- name: PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS
value: "prefect.engine.cloud.CloudFlowRunner"
- name: PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS
value: "prefect.engine.cloud.CloudTaskRunner"
# Custom stuff
- name: CUSTOM__AZ_STORAGE_CONNECTION_STRING
value: "MY CONNECTION STRING"
resources:
requests:
cpu: "100m"
limits:
cpu: "100m"
restartPolicy: Never
2j
10/06/2021, 1:33 PM