Billy McMonagle
01/19/2021, 9:29 PMEnvironment variables
PREFECT__CLOUD__AGENT__AUTH_TOKEN={{undefined}}
PREFECT__CLOUD__API=<https://api.prefect.io>
NAMESPACE=default
IMAGE_PULL_SECRETS=undefined
PREFECT__CLOUD__AGENT__LABELS=['production']
JOB_MEM_REQUEST=undefined
JOB_MEM_LIMIT=undefined
JOB_CPU_REQUEST=undefined
JOB_CPU_LIMIT=undefined
IMAGE_PULL_POLICY=Always
SERVICE_ACCOUNT_NAME=undefined
PREFECT__BACKEND=cloud
PREFECT__CLOUD__AGENT__AGENT_ADDRESS=http://:8080
@task
def execute_query(query):
print("inside execute_query task to run query ", query)
storage = Docker(
registry_url=REGISTRY_URL,
dockerfile=DOCKERFILE,
image_name=IMAGE_NAME,
image_tag=IMAGE_TAG,
)
with Flow("all_accounts", storage=storage) as all_accounts_flow:
execute_query("all_accounts")
storage.build()
if __name__ == "__main__":
all_accounts_flow.register(project_name=PROJECT_NAME)
josh
01/19/2021, 10:02 PM.register
is isn’t updating the image_name:tag in your image repository?Zanie
josh
01/19/2021, 10:05 PMZanie
IMAGE_PULL_POLICY=Always
env var but I’d have to consult to the code to see how that’s templated in the K8s jobsif os.getenv("IMAGE_PULL_POLICY"):
job["spec"]["template"]["spec"]["containers"][0][
"imagePullPolicy"
] = os.getenv("IMAGE_PULL_POLICY")
Billy McMonagle
01/19/2021, 10:31 PMprefect_repo:appname
but it sounds like I should do appname:latest
or something else.IfNotPresent
...Zanie
RunConfig
?Billy McMonagle
01/19/2021, 11:10 PMRunConfig
set currently, although I do intend to use one to set some env vars, once I get over this hump.Zanie
Billy McMonagle
01/19/2021, 11:11 PM│ env: │
│ - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN │
│ valueFrom: │
│ secretKeyRef: │
│ key: prefect_runner_token │
│ name: prefect-orchestration-main │
│ - name: PREFECT__CLOUD__API │
│ value: <https://api.prefect.io> │
│ - name: NAMESPACE │
│ value: default │
│ - name: IMAGE_PULL_SECRETS │
│ - name: PREFECT__CLOUD__AGENT__LABELS │
│ value: '[''production'']' │
│ - name: JOB_MEM_REQUEST │
│ - name: JOB_MEM_LIMIT │
│ - name: JOB_CPU_REQUEST │
│ - name: JOB_CPU_LIMIT │
│ - name: IMAGE_PULL_POLICY │
│ value: Always
Zanie
Billy McMonagle
01/19/2021, 11:12 PMAlways
Zanie
Billy McMonagle
01/19/2021, 11:14 PMZanie
IfNotPresent
in both the Job template and the flow run Pod?Billy McMonagle
01/19/2021, 11:18 PMJob
(I'm using k9s, FWIW)Zanie
KubernetesRun
for image pull policy.Billy McMonagle
01/19/2021, 11:18 PMZanie
Job
for a flow run. The Job
creates a Pod
from its template to actually run the flow. If the Pod
fails, the Job
will create a new one until it succeeds. (https://kubernetes.io/docs/concepts/workloads/controllers/job/)Billy McMonagle
01/19/2021, 11:22 PMZanie
kubectl
? kubectl get jobs
will give you the actual Job
instances. That looks like a Pod
that is named after the job 🙂Billy McMonagle
01/19/2021, 11:24 PM❯ k get jobs
NAME COMPLETIONS DURATION AGE
prefect-job-21ec7c17 0/1 1s 1s
Zanie
kubectl describe job prefect-job-21ec7c17
Billy McMonagle
01/19/2021, 11:25 PMZanie
Billy McMonagle
01/19/2021, 11:26 PM❯ k describe jobs prefect-job-1d696b93
Name: prefect-job-1d696b93
Namespace: default
Selector: controller-uid=b139374d-d47a-4b0f-ab98-18cead0a8eb9
Labels: <http://prefect.io/flow_id=678d4e82-1e48-4d9c-9748-ade3d73867f5|prefect.io/flow_id=678d4e82-1e48-4d9c-9748-ade3d73867f5>
<http://prefect.io/flow_run_id=5e07a5e7-3e49-447d-aba7-47112061db8b|prefect.io/flow_run_id=5e07a5e7-3e49-447d-aba7-47112061db8b>
<http://prefect.io/identifier=1d696b93|prefect.io/identifier=1d696b93>
Annotations: <none>
Parallelism: 1
Completions: 1
Start Time: Tue, 19 Jan 2021 18:25:04 -0500
Completed At: Tue, 19 Jan 2021 18:25:12 -0500
Duration: 8s
Pods Statuses: 0 Running / 1 Succeeded / 0 Failed
Pod Template:
Labels: controller-uid=b139374d-d47a-4b0f-ab98-18cead0a8eb9
job-name=prefect-job-1d696b93
<http://prefect.io/flow_id=678d4e82-1e48-4d9c-9748-ade3d73867f5|prefect.io/flow_id=678d4e82-1e48-4d9c-9748-ade3d73867f5>
<http://prefect.io/flow_run_id=5e07a5e7-3e49-447d-aba7-47112061db8b|prefect.io/flow_run_id=5e07a5e7-3e49-447d-aba7-47112061db8b>
<http://prefect.io/identifier=1d696b93|prefect.io/identifier=1d696b93>
Containers:
flow:
Image: <http://XXX.dkr.ecr.us-east-1.amazonaws.com/prefect-orchestration:grebe|XXX.dkr.ecr.us-east-1.amazonaws.com/prefect-orchestration:grebe>
Port: <none>
Host Port: <none>
Args:
prefect
execute
flow-run
Environment:
PREFECT__CLOUD__API: <https://api.prefect.io>
PREFECT__CLOUD__AUTH_TOKEN: XXX
PREFECT__CLOUD__USE_LOCAL_SECRETS: false
PREFECT__CONTEXT__FLOW_RUN_ID: 5e07a5e7-3e49-447d-aba7-47112061db8b
PREFECT__CONTEXT__FLOW_ID: 678d4e82-1e48-4d9c-9748-ade3d73867f5
PREFECT__CONTEXT__IMAGE: <http://XXX.dkr.ecr.us-east-1.amazonaws.com/prefect-orchestration:grebe|XXX.dkr.ecr.us-east-1.amazonaws.com/prefect-orchestration:grebe>
PREFECT__LOGGING__LOG_TO_CLOUD: true
PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS: prefect.engine.cloud.CloudFlowRunner
PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS: prefect.engine.cloud.CloudTaskRunner
Mounts: <none>
Volumes: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 12s job-controller Created pod: prefect-job-1d696b93-gls7j
Normal Completed 4s job-controller Job complete
Zanie
kubectl get job <HERE> -o yaml | grep imagePull
grep
Billy McMonagle
01/19/2021, 11:31 PMZanie
Billy McMonagle
01/19/2021, 11:33 PMZanie
slugify(pendulum.now("utc").isoformat())
Billy McMonagle
01/19/2021, 11:35 PMZanie
<flow>:<unique-id>
if you want do keep them all in one repo prefect_flows:<flow>-<unique-id>
makes sense to me.Billy McMonagle
01/19/2021, 11:37 PMprefect_flows:<flow>-<unique-id>
convention feels a little bit nicer to me.Zanie
latest
tags — basically the ‘tag’ portion of an image should always be unique and pinned to.Billy McMonagle
01/19/2021, 11:38 PMJim Crist-Harif
01/20/2021, 4:52 PMEnvironment
based configuration.
For flows registered with a run-config (which also includes flows without an explicit flow.environment
set) only options that can be passed to the agent via CLI flags are respected (e.g. --image-pull-secrets
). We didn't elevate every field in the k8s spec, only ones that are likely to be used by most users. To set image_pull_policy
, you'll need to provide a custom k8s job template. This can be be specified via the --job-template
flag, and only needs to include the fields want to add (prefect will set everything it requires). The default job template is
apiVersion: batch/v1
kind: Job
spec:
template:
spec:
containers:
- name: flow
Billy McMonagle
01/20/2021, 4:55 PMJim Crist-Harif
01/20/2021, 5:00 PMenvironment
and run_config
based flows (NAMESPACE
, SERVICE_ACCOUNT_NAME
, and IMAGE_PULL_SECRETS
), but most are already ignored for non-environment
based flows.Billy McMonagle
01/20/2021, 5:36 PM