https://prefect.io logo
Title
m

Michael Ulin

12/17/2021, 12:56 AM
I'm trying to run a workflow using the VertexAgent. I enabled the Vertex API on our GCP account, but I got the following error. Do you know how to set these values?
a

Anna Geller

12/17/2021, 11:36 AM
The error looks like it came from Kubernetes, rather than from Vertex. Can you explain your setup? 1. Do you sue Prefect Cloud or Server? 2. How did you start this Vertex agent? 3. Can you share your run config from the flow that resulted in this error?
m

Michael Ulin

12/17/2021, 3:28 PM
1. Prefect Cloud 2. I ran
prefect vertex agent start
on the command line 3. It's the default vertex run config
flow.run_config = VertexRun()
a

Anna Geller

12/17/2021, 3:33 PM
it looks like either your GCP project, region or service account are not configured on the machine from which you start the VertexAgent - you can either configure those with gcloud CLI or pass those explicitly on the “prefect agent vertex start” as described here
m

Michael Ulin

12/17/2021, 3:41 PM
got it - I'll give it a try
👍 1
thanks!
It looks like the PREFECT__CLOUD__API_KEY and PREFECT__CLOUD__TENANT_ID environment variables are not being set in the Vertex Worker Pool Spec
I'm not sure why though
Oh, I was passing the prefect api key as a token when starting the vertex agent (which had worked with the docker agent)
a

Anna Geller

12/17/2021, 4:22 PM
tokens are deprecated, you should use --key instead
m

Michael Ulin

12/17/2021, 4:23 PM
Yep, that fixed it
thanks for the help!
🙌 1
s

Son Nguyen

01/18/2022, 11:31 AM
sorry for digging this thread again. I’m running into the same errors and just want to confirm that Vertex agent can work with prefect server edition? By adding some logging I can see that some Prefect Cloud envs are being passed when running flow…
👍 1
a

Anna Geller

01/18/2022, 11:49 AM
Good question. In general, all agents should work for both Cloud and Server, even though configuring this for Cloud is certainly much easier due to API keys. I didn’t see any Server user using Vertex agent yet, but I think it should work as long as you configure the Server GraphQL endpoint and GCP service account on the machine from which you start the agent. You can ignore the API key and token in Server, the tenant ID can be set to your default tenant ID
for service account check this and for GraphQL you need to add this to your config.toml:
[server]
endpoint = "<http://YOUR_MACHINES_PUBLIC_IP:4200/graphql|http://YOUR_MACHINES_PUBLIC_IP:4200/graphql>"

  [server.ui]
    apollo_url = "<http://YOUR_MACHINES_PUBLIC_IP:4200/graphql|http://YOUR_MACHINES_PUBLIC_IP:4200/graphql>"
s

Son Nguyen

01/19/2022, 3:26 AM
You can ignore the API key and token in Server
As I understand, when you say
ignore
you mean I don’t need to set the values of these two env vars? If these values are not set, when flow running it’ll throw some errors saying that these two env vars need values.
a

Anna Geller

01/19/2022, 12:49 PM
Did you switch to Server?
prefect backend server
correct, I think those env vars can be left empty/not set for Server
s

Son Nguyen

01/20/2022, 8:50 AM
Did you switch to Server?
I did, but it threw these following errors. I wonder if it’s a bug when using Server backend
400 List of found errors:	1.Field: job_spec.worker_pool_specs[0].container_spec.env[6].value; Message: Required field is not set.	2.Field: job_spec.worker_pool_specs[0].container_spec.env[5].value; Message: Required field is not set.	 [field_violations {
  field: "job_spec.worker_pool_specs[0].container_spec.env[6].value"
  description: "Required field is not set."
}
field_violations {
  field: "job_spec.worker_pool_specs[0].container_spec.env[5].value"
  description: "Required field is not set."
}
]
a

Anna Geller

01/20/2022, 11:02 AM
I wonder if it’s a bug when using Server backend
Does it mean that you tried with Prefect Cloud and it worked, but with Server it didn’t?
s

Son Nguyen

01/21/2022, 3:07 AM
No, I’ve not tried with Prefect Cloud, just had started with Server and it didn’t work.
a

Anna Geller

01/21/2022, 10:16 AM
I could try to reproduce. Can you share more details about what steps did you take? Can you also share the output of the command "prefect diagnostics"?