Hi, I just hosted a prefect 1.2.4 on a gcp vm I am trying to run a flow using VertexRun and I got this error

400 List of found errors:	1.Field: job_spec.worker_pool_specs[0].container_spec.env[12].value; Message: Required field is not set.	2.Field: job_spec.worker_pool_specs[0].container_spec.env[3].value; Message: Required field is not set.	3.Field: job_spec.worker_pool_specs[0].container_spec.env[4].value; Message: Required field is not set.	 [field_violations {
  field: "job_spec.worker_pool_specs[0].container_spec.env[12].value"
  description: "Required field is not set."
}
field_violations {
  field: "job_spec.worker_pool_specs[0].container_spec.env[3].value"
  description: "Required field is not set."
}
field_violations {
  field: "job_spec.worker_pool_specs[0].container_spec.env[4].value"
  description: "Required field is not set."
}
]

Do i need to configure vertex jobs manually?

No, you don't have to. It can be set when starting the agent: https://docs.prefect.io/orchestration/agents/vertex.html

So basically I used the same setup on prefect cloud Running a docker container on the vertex agent Did the same thing on a self hosted instance but I'm getting this error Is this a gcp error or a prefect error?

it looks like you would either need to add more info during the agent start command or provide more info to the VertexRun run config to fill those missing required fields

I looked at a vertex job config for an older flow that i had created using the prefect cloud and i found this config

{
  "workerPoolSpecs": [
    {
      "machineSpec": {
        "machineType": "e2-highmem-8"
      },
      "replicaCount": "1",
      "diskSpec": {
        "bootDiskType": "pd-ssd",
        "bootDiskSizeGb": 100
      },
      "containerSpec": {
        "imageUri": "<image_url> (on dockerhub)",
        "command": [
          "prefect",
          "execute",
          "flow-run"
        ],
        "env": [
          {
            "name": "PREFECT__LOGGING__LEVEL",
            "value": "INFO"
          },
          {
            "name": "PREFECT__BACKEND",
            "value": "cloud"
          },
          {
            "name": "PREFECT__CLOUD__API",
            "value": "<https://api.prefect.io>"
          },
          {
            "name": "PREFECT__CLOUD__API_KEY",
            "value": "<api key>"
          },
          {
            "name": "PREFECT__CLOUD__TENANT_ID",
            "value": "<tenant id>"
          },
          {
            "name": "PREFECT__CLOUD__AGENT__LABELS",
            "value": "['dev']"
          },
          {
            "name": "PREFECT__CLOUD__SEND_FLOW_RUN_LOGS",
            "value": "true"
          },
          {
            "name": "PREFECT__CONTEXT__FLOW_RUN_ID",
            "value": "<flow id>"
          },
          {
            "name": "PREFECT__CONTEXT__FLOW_ID",
            "value": "<flow id>"
          },
          {
            "name": "PREFECT__CLOUD__USE_LOCAL_SECRETS",
            "value": "false"
          },
          {
            "name": "PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS",
            "value": "prefect.engine.cloud.CloudFlowRunner"
          },
          {
            "name": "PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS",
            "value": "prefect.engine.cloud.CloudTaskRunner"
          },
          {
            "name": "PREFECT__LOGGING__LOG_TO_CLOUD",
            "value": "true"
          },
          {
            "name": "PREFECT__CLOUD__AUTH_TOKEN",
            "value": "<prefect auth key>"
          }
        ]
      }
    }
  ],
  "serviceAccount": "<service_account>"
}

The initial error says

[field_violations {
  field: "job_spec.worker_pool_specs[0].container_spec.env[12].value"
  description: "Required field is not set."
}
field_violations {
  field: "job_spec.worker_pool_specs[0].container_spec.env[3].value"
  description: "Required field is not set."
}
field_violations {
  field: "job_spec.worker_pool_specs[0].container_spec.env[4].value"
  description: "Required field is not set."
}
]

env[3] and env[4] are cloud api key and tenant id env[12] is PREFECT_LOGGING_LOG_TO_CLOUD Since i am not using prefect cloud and i have a self hosted instance of prefect 1.2.4 How do i workaround the required fields of cloud api key?

the env variables should look like this (you'd need to adapt it to your JSON spec):

env:
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          value: ''
        - name: PREFECT__CLOUD__API
          value: "<http://some_ip:4200/graphql>" # paste your GraphQL Server endpoint here
        - name: PREFECT__BACKEND
          value: server

I have updated the config to this:

worker_pool_specs = [
        {
            "machine_spec": {"machine_type": "e2-highmem-8"},
            "replica_count": 1,
            "container_spec": {
                "imageUri": "<url to my docker image>",
                "command": [
                    "prefect",
                    "execute",
                    "flow-run"
                ],
                "env": [
                    {
                        "name": "PREFECT__LOGGING__LEVEL",
                        "value": "INFO"
                    },
                    {
                        "name": "PREFECT__BACKEND",
                        "value": "server"
                    },
                    {
                        "name": "PREFECT__CLOUD__API",
                        "value": "http://<external ip to my vm>:4200/graphql"
                    },
                    {
                        "name": "PREFECT__CLOUD__API_KEY",
                        "value": ""
                    },
                    {
                        "name": "PREFECT__CLOUD__TENANT_ID",
                        "value": ""
                    },
                    {
                        "name": "PREFECT__CLOUD__AGENT__LABELS",
                        "value": "['dev']"
                    },
                    {
                        "name": "PREFECT__CLOUD__SEND_FLOW_RUN_LOGS",
                        "value": "true"
                    },
                    {
                        "name": "PREFECT__CONTEXT__FLOW_RUN_ID",
                        "value": ""
                    },
                    {
                        "name": "PREFECT__CONTEXT__FLOW_ID",
                        "value": ""
                    },
                    {
                        "name": "PREFECT__CLOUD__USE_LOCAL_SECRETS",
                        "value": "false"
                    },
                    {
                        "name": "PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS",
                        "value": "prefect.engine.cloud.CloudFlowRunner"
                    },
                    {
                        "name": "PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS",
                        "value": "prefect.engine.cloud.CloudTaskRunner"
                    },
                    {
                        "name": "PREFECT__LOGGING__LOG_TO_CLOUD",
                        "value": ""
                    },
                    {
                        "name": "PREFECT__CLOUD__AUTH_TOKEN",
                        "value": ""
                    }
                ]
            },
        },
    ]

Still getting the same error

not sure why, perhaps for your PoC you can try the same with Prefect Cloud first only to see if this is working and then move to Server orchestration layer once you're sure the execution layer is working?

I have tried the same flow with the same setup on prefect cloud before Everything's the same except In the prefect cloud execution i just passed the machine type and the docker image url as the parameters into the vertex run config

so it works with Cloud but not on Server? if so, could you open an issue in the server repo and describe your issue in detail there?