Shivam Bhatia
07/14/2022, 7:46 AM400 List of found errors: 1.Field: job_spec.worker_pool_specs[0].container_spec.env[12].value; Message: Required field is not set. 2.Field: job_spec.worker_pool_specs[0].container_spec.env[3].value; Message: Required field is not set. 3.Field: job_spec.worker_pool_specs[0].container_spec.env[4].value; Message: Required field is not set. [field_violations {
field: "job_spec.worker_pool_specs[0].container_spec.env[12].value"
description: "Required field is not set."
}
field_violations {
field: "job_spec.worker_pool_specs[0].container_spec.env[3].value"
description: "Required field is not set."
}
field_violations {
field: "job_spec.worker_pool_specs[0].container_spec.env[4].value"
description: "Required field is not set."
}
]
Do i need to configure vertex jobs manually?Anna Geller
07/14/2022, 10:38 AMShivam Bhatia
07/14/2022, 5:22 PMAnna Geller
07/14/2022, 6:32 PMShivam Bhatia
07/15/2022, 8:39 AM{
"workerPoolSpecs": [
{
"machineSpec": {
"machineType": "e2-highmem-8"
},
"replicaCount": "1",
"diskSpec": {
"bootDiskType": "pd-ssd",
"bootDiskSizeGb": 100
},
"containerSpec": {
"imageUri": "<image_url> (on dockerhub)",
"command": [
"prefect",
"execute",
"flow-run"
],
"env": [
{
"name": "PREFECT__LOGGING__LEVEL",
"value": "INFO"
},
{
"name": "PREFECT__BACKEND",
"value": "cloud"
},
{
"name": "PREFECT__CLOUD__API",
"value": "<https://api.prefect.io>"
},
{
"name": "PREFECT__CLOUD__API_KEY",
"value": "<api key>"
},
{
"name": "PREFECT__CLOUD__TENANT_ID",
"value": "<tenant id>"
},
{
"name": "PREFECT__CLOUD__AGENT__LABELS",
"value": "['dev']"
},
{
"name": "PREFECT__CLOUD__SEND_FLOW_RUN_LOGS",
"value": "true"
},
{
"name": "PREFECT__CONTEXT__FLOW_RUN_ID",
"value": "<flow id>"
},
{
"name": "PREFECT__CONTEXT__FLOW_ID",
"value": "<flow id>"
},
{
"name": "PREFECT__CLOUD__USE_LOCAL_SECRETS",
"value": "false"
},
{
"name": "PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS",
"value": "prefect.engine.cloud.CloudFlowRunner"
},
{
"name": "PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS",
"value": "prefect.engine.cloud.CloudTaskRunner"
},
{
"name": "PREFECT__LOGGING__LOG_TO_CLOUD",
"value": "true"
},
{
"name": "PREFECT__CLOUD__AUTH_TOKEN",
"value": "<prefect auth key>"
}
]
}
}
],
"serviceAccount": "<service_account>"
}
The initial error says
[field_violations {
field: "job_spec.worker_pool_specs[0].container_spec.env[12].value"
description: "Required field is not set."
}
field_violations {
field: "job_spec.worker_pool_specs[0].container_spec.env[3].value"
description: "Required field is not set."
}
field_violations {
field: "job_spec.worker_pool_specs[0].container_spec.env[4].value"
description: "Required field is not set."
}
]
env[3] and env[4] are cloud api key and tenant id
env[12] is PREFECT_LOGGING_LOG_TO_CLOUD
Since i am not using prefect cloud and i have a self hosted instance of prefect 1.2.4
How do i workaround the required fields of cloud api key?Anna Geller
07/15/2022, 4:49 PMenv:
- name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
value: ''
- name: PREFECT__CLOUD__API
value: "<http://some_ip:4200/graphql>" # paste your GraphQL Server endpoint here
- name: PREFECT__BACKEND
value: server
PREFECT__CLOUD__API_KEY
can also be set to an empty stringShivam Bhatia
07/16/2022, 5:58 AMworker_pool_specs = [
{
"machine_spec": {"machine_type": "e2-highmem-8"},
"replica_count": 1,
"container_spec": {
"imageUri": "<url to my docker image>",
"command": [
"prefect",
"execute",
"flow-run"
],
"env": [
{
"name": "PREFECT__LOGGING__LEVEL",
"value": "INFO"
},
{
"name": "PREFECT__BACKEND",
"value": "server"
},
{
"name": "PREFECT__CLOUD__API",
"value": "http://<external ip to my vm>:4200/graphql"
},
{
"name": "PREFECT__CLOUD__API_KEY",
"value": ""
},
{
"name": "PREFECT__CLOUD__TENANT_ID",
"value": ""
},
{
"name": "PREFECT__CLOUD__AGENT__LABELS",
"value": "['dev']"
},
{
"name": "PREFECT__CLOUD__SEND_FLOW_RUN_LOGS",
"value": "true"
},
{
"name": "PREFECT__CONTEXT__FLOW_RUN_ID",
"value": ""
},
{
"name": "PREFECT__CONTEXT__FLOW_ID",
"value": ""
},
{
"name": "PREFECT__CLOUD__USE_LOCAL_SECRETS",
"value": "false"
},
{
"name": "PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS",
"value": "prefect.engine.cloud.CloudFlowRunner"
},
{
"name": "PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS",
"value": "prefect.engine.cloud.CloudTaskRunner"
},
{
"name": "PREFECT__LOGGING__LOG_TO_CLOUD",
"value": ""
},
{
"name": "PREFECT__CLOUD__AUTH_TOKEN",
"value": ""
}
]
},
},
]
Still getting the same errorAnna Geller
07/16/2022, 12:27 PMShivam Bhatia
07/16/2022, 1:55 PMAnna Geller
07/16/2022, 4:41 PM