Paco Ibañez
10/28/2022, 2:23 PMChristopher Boyd
10/28/2022, 3:08 PMPaco Ibañez
10/28/2022, 3:20 PMFinished in state TimedOut('Flow run exceeded timeout of 1800.0 seconds
. the nodes are 8 cores 32 Gb. The flow calls a DS model to get some predictions and it looks like that is what it is taking most of the time. the model is ~500 MbChristopher Boyd
10/28/2022, 3:37 PMPaco Ibañez
10/28/2022, 3:39 PMChristopher Boyd
10/28/2022, 3:49 PMPaco Ibañez
10/28/2022, 4:12 PMcustomizations=[{
"op": "add",
"path": "/spec/template/spec/resources",
"value": {"requests": {"memory": "2Gi", "cpu": "2"}},
}]
infrastructure = KubernetesJob(
image = image,
customizations = customizations,
finished_job_ttl = 1*60*60, # one hour
)
k8s_job = KubernetesJob.base_job_manifest()
k8s_job['spec']['template']['spec']['resources'] = {"requests": {"memory": "8Gi", "cpu": "2"}}
infrastructure = KubernetesJob(
image = image,
job = k8s_job,
Christopher Boyd
10/28/2022, 4:29 PMPaco Ibañez
10/28/2022, 4:30 PMChristopher Boyd
10/28/2022, 5:08 PMcustomizations=[
{
"op": "add",
"path": "/spec/imagePullSecrets",
"value": [{'name': 'dockerhub'}],
},
{
"op": "add",
"path": "/spec/template/spec/resources",
"value": {"limits": {"memory": "8Gi", "cpu": "4000m"}},
}
],
spec:
template:
spec:
completions: 1
containers: # the first container is required
- env: []
name: prefect-job
image: prefecthq/prefect:2.3.0-python3.9
imagePullPolicy: "IfNotPresent"
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
parallelism: 1
restartPolicy: Never
Paco Ibañez
10/28/2022, 5:11 PM