I am trying to transition our deployments from a C...
# prefect-gcp
i
I am trying to transition our deployments from a Cloud Run worker pool to a Cloud Run V2 worker pool in order to leverage the longer timeouts >3600s. I have been doing testing but have had issues getting it to work. Any time I run a deployment on the worker pool I get the following error:
Copy code
Flow run could not be submitted to infrastructure: <https://run.googleapis.com/v2/projects/trading-system-dev/locations/us-central1/jobs?jobId=prefect-cyber-sparrow&alt=json> returned "Violation in CreateJobRequest.job.template.template.vpc_access.connector: one and only one of connector and network_interfaces must be set.". Details: "[{'@type': '<http://type.googleapis.com/google.rpc.BadRequest|type.googleapis.com/google.rpc.BadRequest>', 'fieldViolations': [{'field': 'Violation in CreateJobRequest.job.template.template.vpc_access.connector', 'description': 'one and only one of connector and network_interfaces must be set.'}]}]">
From what i've gathered, prefect uses Terraform and cloud run seems to require either a connector or network_interfaces. I am not specifying a vpc connector (which prefect has marked as optional). It almost seems as though when there is no vpc connector, the network_interfaces should be used as a default but is not happening. Any help or advice would be greatly appreciated.
j
I’ll take a look when I’m back at computer. What version of prefect-GCP are you utilizing ?
i
I was using 0.4.7. I just tried upgrading that. the thought never occurred to check that as I just assumed it had something to do with the work pool which I created on the frontend. I'll post an update once i've tested with the updated prefect-gcp
🙏 1
I just updated prefect-gcp and redeployed a deployment on the new worker pool using cloud run v2 and unfortunately, I am still seeing the same error... let me know if there is any other information I can provide
j
Darn :( I’ll ping thread in 20 mins when I’m back
Hey @Isaac, push or pull worker?
i
push
j
and would you share base job template JSON from Work Pool
make sure there's no PII or anything too before posting
i
Copy code
{
  "variables": {
    "type": "object",
    "required": [
      "credentials"
    ],
    "properties": {
      "cpu": {
        "type": "string",
        "title": "CPU",
        "default": "1000m",
        "pattern": "^(\\d*000)m$",
        "description": "The CPU to allocate to the Cloud Run job."
      },
      "env": {
        "type": "object",
        "title": "Environment Variables",
        "default": {
          XXX
        },
        "description": "Environment variables to set when starting a flow run.",
        "additionalProperties": {
          "type": "string"
        }
      },
      "args": {
        "type": "array",
        "items": {
          "type": "string"
        },
        "title": "Args",
        "description": "The arguments to pass to the Cloud Run Job V2's entrypoint command."
      },
      "name": {
        "type": "string",
        "title": "Name",
        "description": "Name given to created infrastructure."
      },
      "image": {
        "type": "string",
        "title": "Image Name",
        "default": "<http://docker.io/prefecthq/prefect:2-latest|docker.io/prefecthq/prefect:2-latest>",
        "description": "The image to use for the Cloud Run job. If not provided the latest Prefect image will be used."
      },
      "labels": {
        "type": "object",
        "title": "Labels",
        "description": "Labels applied to created infrastructure.",
        "additionalProperties": {
          "type": "string"
        }
      },
      "memory": {
        "type": "string",
        "title": "Memory",
        "default": "512Mi",
        "example": "512Mi",
        "pattern": "^\\d+(?:G|Gi|M|Mi)$",
        "description": "The memory to allocate to the Cloud Run job along with the units, which could be: G, Gi, M, Mi."
      },
      "region": {
        "type": "string",
        "title": "Region",
        "default": "us-central1",
        "description": "The region in which to run the Cloud Run job"
      },
      "command": {
        "type": "string",
        "title": "Command",
        "description": "The command to use when starting a flow run. In most cases, this should be left blank and the command will be automatically generated."
      },
      "timeout": {
        "type": "integer",
        "title": "Job Timeout",
        "default": 10800,
        "maximum": 86400,
        "description": "The length of time that Prefect will wait for a Cloud Run Job to complete before raising an exception (maximum of 86400 seconds, 1 day).",
        "exclusiveMinimum": 0
      },
      "keep_job": {
        "type": "boolean",
        "title": "Keep Job After Completion",
        "default": false,
        "description": "Keep the completed Cloud run job on Google Cloud Platform."
      },
      "credentials": {
        "allOf": [
          {
            "$ref": "#/definitions/GcpCredentials"
          }
        ],
        "title": "GCP Credentials",
        "default": {
          "$ref": {
            "block_document_id": "5d3f0458-e59b-4c44-a481-c174fd3768b5"
          }
        },
        "description": "The GCP Credentials used to initiate the Cloud Run Job. `service_account_info` is required."
      },
      "max_retries": {
        "type": "integer",
        "title": "Max Retries",
        "default": 0,
        "description": "The number of times to retry the Cloud Run job."
      },
      "launch_stage": {
        "enum": [
          "ALPHA",
          "BETA",
          "GA",
          "DEPRECATED",
          "EARLY_ACCESS",
          "PRELAUNCH",
          "UNIMPLEMENTED",
          "LAUNCH_TAG_UNSPECIFIED"
        ],
        "type": "string",
        "title": "Launch Stage",
        "default": "BETA",
        "description": "The launch stage of the Cloud Run Job V2. See <https://cloud.google.com/run/docs/about-features-categories> for additional details."
      },
      "vpc_connector_name": {
        "type": "string",
        "title": "VPC Connector Name",
        "description": "The name of the VPC connector to use for the Cloud Run job."
      },
      "service_account_name": {
        "type": "string",
        "title": "Service Account Name",
        "example": "<mailto:service-account@example.iam.gserviceaccount.com|service-account@example.iam.gserviceaccount.com>",
        "description": "The name of the service account to use for the task execution of Cloud Run Job. By default Cloud Run jobs run as the default Compute Engine Service Account. "
      }
    },
    "definitions": {
      "GcpCredentials": {
        "type": "object",
        "title": "GcpCredentials",
        "properties": {
          "project": {
            "type": "string",
            "title": "Project",
            "description": "The GCP project to use for the client."
          },
          "service_account_file": {
            "type": "string",
            "title": "Service Account File",
            "format": "path",
            "description": "Path to the service account JSON keyfile."
          },
          "service_account_info": {
            "type": "object",
            "title": "Service Account Info",
            "description": "The contents of the keyfile as a dict."
          }
        },
        "description": "Block used to manage authentication with GCP. Google authentication is handled via the `google.oauth2.\nSee Google's [Authentication documentation](<https://cloud.google.com/docs/authentication#service-accounts>)\nfor details on inference and recommended authentication patterns.",
        "secret_fields": [
          "service_account_info.*"
        ],
        "block_type_slug": "gcp-credentials",
        "block_schema_references": {}
      }
    },
    "description": "Default variables for the Cloud Run V2 Push Pool.\n\nThe schema for this class is used to populate the `variables` section of the\ndefault base job template."
  },
  "job_configuration": {
    "env": "{{ env }}",
    "name": "{{ name }}",
    "labels": "{{ labels }}",
    "region": "{{ region }}",
    "command": "{{ command }}",
    "timeout": "{{ timeout }}",
    "job_body": {
      "client": "prefect",
      "template": {
        "template": {
          "timeout": "{{ timeout }}",
          "vpcAccess": {
            "connector": "{{ vpc_connector_name }}"
          },
          "containers": [
            {
              "env": [],
              "args": "{{ args }}",
              "image": "{{ image }}",
              "command": "{{ command }}",
              "resources": {
                "limits": {
                  "cpu": "{{ cpu }}",
                  "memory": "{{ memory }}"
                }
              }
            }
          ],
          "maxRetries": "{{ max_retries }}",
          "serviceAccount": "{{ service_account_name }}"
        }
      },
      "launchStage": "{{ launch_stage }}"
    },
    "keep_job": "{{ keep_job }}",
    "credentials": "{{ credentials }}"
  }
}
👀 1
j
It looks like there was an issue resolved today around this topic, here. @Nate / @Jake Kaplan, sorry to bother you at off hours/saw you both on PR and you both have a better understanding of Push Workers release cycle than me, is this fix released yet to Push Workers?
n
hey @Jack P - let me look
❤️ 1
i
as for the defaults, the only things that I changed were the timeouts, added in env variables, and set the GCP credentials. the credentials and environment variables were the same that I use in our v1 cloud run push pool.
n
it does seem you're running into the issue solved in #267, which jake just made a corresponding fix for in cloud and released into the world which should fix that for push work pools
🐐 1
ill get a patch release out for
prefect-gcp
so its also fixed in hybrid pools asap
🙌 1
j
Thanks Nate, sorry about introducing that bug 🐛 😬 Let me know if you need any assistance!
n
no sweat! it happens - I released
marvin
not too long ago with a literal
breakpoint()
in there someone shared their traceback and I was like, "oh where are you calling that breakpoint?" and they were like "I'm not" 🙃
🚀 1
i
Thanks @Nate @Jack P for taking a look. look like this resolved the issue and will be a major help for us going forwards!
catjam 2
100 Views