https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> - I have the below config for a googl...
# ask-marvin
r
@Marvin - I have the below config for a google cloud run worker, and I am having issues with the "prefect_server_api_auth_string_secret" not being passed properly into the instance that is created when a run is triggered. It works fine for the "prefect_api_auth_string_secret" - please help me figure out what is wrong with the config: (Top is left out)
Copy code
"image": {
        "anyOf": [
          {
            "type": "string"
          },
          {
            "type": "null"
          }
        ],
        "title": "Image Name",
        "default": "prefecthq/prefect:3-latest",
        "description": "The image to use for the Cloud Run job. If not provided the default Prefect image will be used."
      },
      "labels": {
        "type": "object",
        "title": "Labels",
        "description": "Labels applied to infrastructure created by a worker.",
        "additionalProperties": {
          "type": "string"
        }
      },
      "memory": {
        "type": "string",
        "title": "Memory",
        "default": "512Mi",
        "pattern": "^\\d+(?:G|Gi|M|Mi)$",
        "examples": [
          "512Mi"
        ],
        "description": "The memory to allocate to the Cloud Run job along with the units, whichcould be: G, Gi, M, Mi."
      },
      "region": {
        "type": "string",
        "title": "Region",
        "default": "europe-north2",
        "description": "The region in which to run the Cloud Run job"
      },
      "command": {
        "anyOf": [
          {
            "type": "string"
          },
          {
            "type": "null"
          }
        ],
        "title": "Command",
        "description": "The command to use when starting a flow run. In most cases, this should be left blank and the command will be automatically generated by the worker."
      },
      "timeout": {
        "type": "integer",
        "title": "Job Timeout",
        "default": 600,
        "maximum": 86400,
        "description": "Max allowed time duration the Job may be active before Cloud Run will  actively try to mark it failed and kill associated containers (maximum of 86400 seconds, 1 day).",
        "exclusiveMinimum": 0
      },
      "keep_job": {
        "type": "boolean",
        "title": "Keep Job After Completion",
        "default": false,
        "description": "Keep the completed Cloud run job on Google Cloud Platform."
      },
      "credentials": {
        "$ref": "#/definitions/GcpCredentials",
        "title": "GCP Credentials",
        "description": "The GCP Credentials used to connect to Cloud Run. If not provided credentials will be inferred from the local environment."
      },
      "max_retries": {
        "type": "integer",
        "title": "Max Retries",
        "default": 0,
        "description": "The number of times to retry the Cloud Run job."
      },
      "launch_stage": {
        "enum": [
          "ALPHA",
          "BETA",
          "GA",
          "DEPRECATED",
          "EARLY_ACCESS",
          "PRELAUNCH",
          "UNIMPLEMENTED",
          "LAUNCH_TAG_UNSPECIFIED"
        ],
        "type": "string",
        "title": "Launch Stage",
        "default": "BETA",
        "description": "The launch stage of the Cloud Run Job V2. See <https://cloud.google.com/run/docs/about-features-categories> for additional details."
      },
      "env_from_secrets": {
        "type": "object",
        "title": "Environment Variables from Secrets",
        "examples": [
          {
            "ENV_VAR_NAME": {
              "secret": "SECRET_NAME",
              "version": "latest"
            }
          }
        ],
        "description": "Environment variables to set from GCP secrets when starting a flow run.",
        "additionalProperties": {
          "$ref": "#/definitions/SecretKeySelector"
        }
      },
      "cloudsql_instances": {
        "anyOf": [
          {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          {
            "type": "null"
          }
        ],
        "title": "Cloud SQL Instances",
        "default": [
          "infrastructure-467509:europe-north2:primary"
        ],
        "examples": [
          "project:region:instance-id"
        ],
        "description": "List of Cloud SQL instance connection names to connect to. Format: {project}:{location}:{instance}"
      },
      "vpc_connector_name": {
        "anyOf": [
          {
            "type": "string"
          },
          {
            "type": "null"
          }
        ],
        "title": "VPC Connector Name",
        "description": "The name of the VPC connector to use for the Cloud Run job."
      },
      "service_account_name": {
        "anyOf": [
          {
            "type": "string"
          },
          {
            "type": "null"
          }
        ],
        "title": "Service Account Name",
        "default": "<mailto:prefect-worker-sa@infrastructure-467509.iam.gserviceaccount.com|prefect-worker-sa@infrastructure-467509.iam.gserviceaccount.com>",
        "examples": [
          "<mailto:service-account@example.iam.gserviceaccount.com|service-account@example.iam.gserviceaccount.com>"
        ],
        "description": "The name of the service account to use for the task execution of Cloud Run Job. By default Cloud Run jobs run as the default Compute Engine Service Account."
      },
      "prefect_api_key_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect API Key Secret",
        "examples": [
          {
            "secret": "prefect-api-key",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect API key. When provided, the secret will be used instead of the PREFECT_API_KEY environment variable."
      },
      "prefect_api_auth_string_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect API Auth String Secret",
        "default": {
          "secret": "prefect-api-auth",
          "version": "latest"
        },
        "examples": [
          {
            "secret": "prefect-api-auth-string",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect API auth string. When provided, the secret will be used instead of the PREFECT_API_AUTH_STRING environment variable."
      },
      "prefect_server_api_auth_string_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect Server API Auth String Secret",
        "default": {
          "secret": "prefect-api-auth",
          "version": "latest"
        },
        "examples": [
          {
            "secret": "prefect-server-api-auth-string",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect Server API auth string. When provided, the secret will be used instead of the PREFECT_SERVER_API_AUTH_STRING environment variable."
      }
    },
    "definitions": {
      "GcpCredentials": {
        "type": "object",
        "title": "GcpCredentials",
        "properties": {
          "project": {
            "anyOf": [
              {
                "type": "string"
              },
              {
                "type": "null"
              }
            ],
            "title": "Project",
            "default": null,
            "description": "The GCP project to use for the client."
          },
          "service_account_file": {
            "anyOf": [
              {
                "type": "string",
                "format": "path"
              },
              {
                "type": "null"
              }
            ],
            "title": "Service Account File",
            "default": null,
            "description": "Path to the service account JSON keyfile."
          },
          "service_account_info": {
            "anyOf": [
              {
                "type": "object",
                "additionalProperties": true
              },
              {
                "type": "null"
              }
            ],
            "title": "Service Account Info",
            "default": null,
            "description": "The contents of the keyfile as a dict."
          }
        },
        "description": "Block used to manage authentication with GCP. Google authentication is\nhandled via the `google.oauth2` module or through the CLI.\nSpecify either one of service `account_file` or `service_account_info`; if both\nare not specified, the client will try to detect the credentials following Google's\n[Application Default Credentials](<https://cloud.google.com/docs/authentication/application-default-credentials>).\nSee Google's [Authentication documentation](<https://cloud.google.com/docs/authentication#service-accounts>)\nfor details on inference and recommended authentication patterns.",
        "secret_fields": [
          "service_account_info.*"
        ],
        "block_type_slug": "gcp-credentials",
        "block_schema_references": {}
      },
      "SecretKeySelector": {
        "type": "object",
        "title": "SecretKeySelector",
        "required": [
          "secret",
          "version"
        ],
        "properties": {
          "secret": {
            "type": "string",
            "title": "Secret"
          },
          "version": {
            "type": "string",
            "title": "Version"
          }
        },
        "description": "SecretKeySelector is a data model for specifying a GCP secret to inject\ninto a Cloud Run V2 Job as an environment variable.\nFollows Cloud Run V2 rest API, docs:\n<https://cloud.google.com/run/docs/reference/rest/v2/Container#SecretKeySelector>"
      }
    },
    "description": "Default variables for the v2 Cloud Run worker.\n\nThe schema for this class is used to populate the `variables` section of the\ndefault base job template."
  },
  "job_configuration": {
    "env": "{{ env }}",
    "name": "{{ name }}",
    "labels": "{{ labels }}",
    "region": "{{ region }}",
    "command": "{{ command }}",
    "timeout": "{{ timeout }}",
    "job_body": {
      "client": "prefect",
      "template": {
        "template": {
          "timeout": "{{ timeout }}",
          "vpcAccess": {
            "connector": "{{ vpc_connector_name }}"
          },
          "containers": [
            {
              "env": [],
              "args": "{{ args }}",
              "image": "{{ image }}",
              "command": "{{ command }}",
              "resources": {
                "limits": {
                  "cpu": "{{ cpu }}",
                  "memory": "{{ memory }}"
                }
              }
            }
          ],
          "maxRetries": "{{ max_retries }}",
          "serviceAccount": "{{ service_account_name }}"
        }
      },
      "launchStage": "{{ launch_stage }}"
    },
    "keep_job": "{{ keep_job }}",
    "credentials": "{{ credentials }}",
    "env_from_secrets": "{{ env_from_secrets }}",
    "cloudsql_instances": "{{ cloudsql_instances }}",
    "prefect_api_key_secret": "{{ prefect_api_key_secret }}",
    "prefect_api_auth_string_secret": "{{ prefect_api_auth_string_secret }}",
    "prefect_server_api_auth_string_secret": "{{ prefect_server_api_auth_string_secret }}"
  }
}
m
Your message was too long, here's your message at the allowed limit: - I have the below config for a google cloud run worker and I am having issues with the prefect_server_api_auth_string_secret not being passed properly into the instance that is created when a run is triggered It works fine for the prefect_api_auth_string_secret - please help me figure out what is wrong with the config (Top is left out) ``` image { anyOf [ { type string } { type null } ] title Image Name default prefecthq/prefect 3-latest description The image to use for the Cloud Run job If not provided the default Prefect image will be used } labels { type object title Labels description Labels applied to infrastructure created by a worker additionalProperties { type string } } memory { type string title Memory default 512Mi pattern ^\\d+(? G|Gi|M|Mi)$ examples [ 512Mi ] description The memory to allocate to the Cloud Run job along with the units whichcould be G Gi M Mi } region { type string title Region default europe-north2 description The region in which to run the Cloud Run job } command { anyOf [ { type string } { type null } ] title Command description The command to use when starting a flow run In most cases this should be left blank and the command will be automatically generated by the worker } timeout { type integer title Job Timeout default 600 maximum 86400 description Max allowed time duration the Job may be active before Cloud Run will actively try to mark it failed and kill associated containers (maximum of 86400 seconds 1 day) exclusiveMinimum 0 } keep_job { type boolean title Keep Job After Completion default false description Keep the completed Cloud run job on Google Cloud Platform } credentials { $ref #/definitions/GcpCredentials title GCP Credentials description The GCP Credentials used to connect to Cloud Run If not provided credentials will be inferred from the local environment } max_retries { type integer title Max Retries default 0 description The number of times to retry the Cloud Run job } launch_stage { enum [ ALPHA BETA GA DEPRECATED EARLY_ACCESS PRELAUNCH UNIMPLEMENTED LAUNCH_TAG_UNSPECIFIED ] type string title Launch Stage default BETA description The launch stage of the Cloud Run Job V2 See <https //cloud google com/run/docs/about-features-categories> for additional details } env_from_secrets { type object title Environment Variables from Secrets examples [ { ENV_VAR_NAME { secret SECRET_NAME version latest } } ] description Environment variables to set from GCP secrets when starting a flow run additionalProperties { $ref #/definitions/SecretKeySelector } } cloudsql_instances { anyOf [ { type array items { type string } } { type null } ] title Cloud SQL Instances default [ infrastructure-467509 europe-north2 primary ] examples [ project region instance-id ] description List of Cloud SQL instance connection names to connect to Format {project} {location} {instance} } vpc_connector_name { anyOf [ { type string } { type null } ] title VPC Connector Name description The name of the VPC connector to use for the Cloud Run job } service_account_name { anyOf
r
@Marvin - I have the below config for a google cloud run worker, and I am having issues with the "prefect_server_api_auth_string_secret" not being passed properly into the instance that is created when a run is triggered. It works fine for the "prefect_api_auth_string_secret" - please help me figure out what is wrong with the config:
Copy code
"env_from_secrets": {
        "type": "object",
        "title": "Environment Variables from Secrets",
        "examples": [
          {
            "ENV_VAR_NAME": {
              "secret": "SECRET_NAME",
              "version": "latest"
            }
          }
        ],
        "description": "Environment variables to set from GCP secrets when starting a flow run.",
        "additionalProperties": {
          "$ref": "#/definitions/SecretKeySelector"
        }
      },
      "cloudsql_instances": {
        "anyOf": [
          {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          {
            "type": "null"
          }
        ],
        "title": "Cloud SQL Instances",
        "default": [
          "infrastructure-467509:europe-north2:primary"
        ],
        "examples": [
          "project:region:instance-id"
        ],
        "description": "List of Cloud SQL instance connection names to connect to. Format: {project}:{location}:{instance}"
      },
      "vpc_connector_name": {
        "anyOf": [
          {
            "type": "string"
          },
          {
            "type": "null"
          }
        ],
        "title": "VPC Connector Name",
        "description": "The name of the VPC connector to use for the Cloud Run job."
      },
      "service_account_name": {
        "anyOf": [
          {
            "type": "string"
          },
          {
            "type": "null"
          }
        ],
        "title": "Service Account Name",
        "default": "<mailto:prefect-worker-sa@infrastructure-467509.iam.gserviceaccount.com|prefect-worker-sa@infrastructure-467509.iam.gserviceaccount.com>",
        "examples": [
          "<mailto:service-account@example.iam.gserviceaccount.com|service-account@example.iam.gserviceaccount.com>"
        ],
        "description": "The name of the service account to use for the task execution of Cloud Run Job. By default Cloud Run jobs run as the default Compute Engine Service Account."
      },
      "prefect_api_key_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect API Key Secret",
        "examples": [
          {
            "secret": "prefect-api-key",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect API key. When provided, the secret will be used instead of the PREFECT_API_KEY environment variable."
      },
      "prefect_api_auth_string_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect API Auth String Secret",
        "default": {
          "secret": "prefect-api-auth",
          "version": "latest"
        },
        "examples": [
          {
            "secret": "prefect-api-auth-string",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect API auth string. When provided, the secret will be used instead of the PREFECT_API_AUTH_STRING environment variable."
      },
      "prefect_server_api_auth_string_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect Server API Auth String Secret",
        "default": {
          "secret": "prefect-api-auth",
          "version": "latest"
        },
        "examples": [
          {
            "secret": "prefect-server-api-auth-string",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect Server API auth string. When provided, the secret will be used instead of the PREFECT_SERVER_API_AUTH_STRING environment variable."
      }
    },
    "definitions": {
      "GcpCredentials": {
        "type": "object",
        "title": "GcpCredentials",
        "properties": {
          "project": {
            "anyOf": [
              {
                "type": "string"
              },
              {
                "type": "null"
              }
            ],
            "title": "Project",
            "default": null,
            "description": "The GCP project to use for the client."
          },
          "service_account_file": {
            "anyOf": [
              {
                "type": "string",
                "format": "path"
              },
              {
                "type": "null"
              }
            ],
            "title": "Service Account File",
            "default": null,
            "description": "Path to the service account JSON keyfile."
          },
          "service_account_info": {
            "anyOf": [
              {
                "type": "object",
                "additionalProperties": true
              },
              {
                "type": "null"
              }
            ],
            "title": "Service Account Info",
            "default": null,
            "description": "The contents of the keyfile as a dict."
          }
        },
        "description": "Block used to manage authentication with GCP. Google authentication is\nhandled via the `google.oauth2` module or through the CLI.\nSpecify either one of service `account_file` or `service_account_info`; if both\nare not specified, the client will try to detect the credentials following Google's\n[Application Default Credentials](<https://cloud.google.com/docs/authentication/application-default-credentials>).\nSee Google's [Authentication documentation](<https://cloud.google.com/docs/authentication#service-accounts>)\nfor details on inference and recommended authentication patterns.",
        "secret_fields": [
          "service_account_info.*"
        ],
        "block_type_slug": "gcp-credentials",
        "block_schema_references": {}
      },
      "SecretKeySelector": {
        "type": "object",
        "title": "SecretKeySelector",
        "required": [
          "secret",
          "version"
        ],
        "properties": {
          "secret": {
            "type": "string",
            "title": "Secret"
          },
          "version": {
            "type": "string",
            "title": "Version"
          }
        },
        "description": "SecretKeySelector is a data model for specifying a GCP secret to inject\ninto a Cloud Run V2 Job as an environment variable.\nFollows Cloud Run V2 rest API, docs:\n<https://cloud.google.com/run/docs/reference/rest/v2/Container#SecretKeySelector>"
      }
    },
    "description": "Default variables for the v2 Cloud Run worker.\n\nThe schema for this class is used to populate the `variables` section of the\ndefault base job template."
  },
  "job_configuration": {
    "env": "{{ env }}",
    "name": "{{ name }}",
    "labels": "{{ labels }}",
    "region": "{{ region }}",
    "command": "{{ command }}",
    "timeout": "{{ timeout }}",
    "job_body": {
      "client": "prefect",
      "template": {
        "template": {
          "timeout": "{{ timeout }}",
          "vpcAccess": {
            "connector": "{{ vpc_connector_name }}"
          },
          "containers": [
            {
              "env": [],
              "args": "{{ args }}",
              "image": "{{ image }}",
              "command": "{{ command }}",
              "resources": {
                "limits": {
                  "cpu": "{{ cpu }}",
                  "memory": "{{ memory }}"
                }
              }
            }
          ],
          "maxRetries": "{{ max_retries }}",
          "serviceAccount": "{{ service_account_name }}"
        }
      },
      "launchStage": "{{ launch_stage }}"
    },
    "keep_job": "{{ keep_job }}",
    "credentials": "{{ credentials }}",
    "env_from_secrets": "{{ env_from_secrets }}",
    "cloudsql_instances": "{{ cloudsql_instances }}",
    "prefect_api_key_secret": "{{ prefect_api_key_secret }}",
    "prefect_api_auth_string_secret": "{{ prefect_api_auth_string_secret }}",
    "prefect_server_api_auth_string_secret": "{{ prefect_server_api_auth_string_secret }}"
  }
}
First part of the config is left out:
m
Your message was too long, here's your message at the allowed limit: - I have the below config for a google cloud run worker and I am having issues with the prefect_server_api_auth_string_secret not being passed properly into the instance that is created when a run is triggered It works fine for the prefect_api_auth_string_secret - please help me figure out what is wrong with the config ``` env_from_secrets { type object title Environment Variables from Secrets examples [ { ENV_VAR_NAME { secret SECRET_NAME version latest } } ] description Environment variables to set from GCP secrets when starting a flow run additionalProperties { $ref #/definitions/SecretKeySelector } } cloudsql_instances { anyOf [ { type array items { type string } } { type null } ] title Cloud SQL Instances default [ infrastructure-467509 europe-north2 primary ] examples [ project region instance-id ] description List of Cloud SQL instance connection names to connect to Format {project} {location} {instance} } vpc_connector_name { anyOf [ { type string } { type null } ] title VPC Connector Name description The name of the VPC connector to use for the Cloud Run job } service_account_name { anyOf [ { type string } { type null } ] title Service Account Name default <mailto prefect-worker-sa@infrastructure-467509 iam gserviceaccount com|prefect-worker-sa@infrastructure-467509 iam gserviceaccount com> examples [ <mailto service-account@example iam gserviceaccount com|service-account@example iam gserviceaccount com> ] description The name of the service account to use for the task execution of Cloud Run Job By default Cloud Run jobs run as the default Compute Engine Service Account } prefect_api_key_secret { anyOf [ { $ref #/definitions/SecretKeySelector } { type null } ] title Prefect API Key Secret examples [ { secret prefect-api-key version latest } ] description The GCP secret to use for the Prefect API key When provided the secret will be used instead of the PREFECT_API_KEY environment variable } prefect_api_auth_string_secret { anyOf [ { $ref #/definitions/SecretKeySelector } { type null } ] title Prefect API Auth String Secret default { secret prefect-api-auth version latest } examples [ { secret prefect-api-auth-string version latest } ] description The GCP secret to use for the Prefect API auth string When provided the secret will be used instead of the PREFECT_API_AUTH_STRING environment variable } prefect_server_api_auth_string_secret { anyOf [ { $ref #/definitions/SecretKeySelector } { type null } ] title Prefect Server API Auth String Secret default { secret prefect-api-auth version latest } examples [ { secret prefect-server-api-auth-string version latest } ] description The GCP secret to use for the Prefect Server API auth string When provided the secret will be used instead of the PREFECT_SERVER_API_AUTH_STRING environment variable } } definitions { GcpCredentials { type object title GcpCredentials properties { project { anyOf [ { type string } { type null } ] title Project default null description The GCP project to use for the client } service_account_file { anyOf [ { type string format path } { type null } ] title Service Account File default null description Path to the service account JSON keyfile } service_account_info { anyOf [ { type object additionalProperties true } { type
r
@Marvin - I have the below config for a google cloud run worker, and I am having issues with the "prefect_server_api_auth_string_secret" not being passed properly into the instance that is created when a run is triggered. It works fine for the "prefect_api_auth_string_secret" - please help me figure out what is wrong with the config:
Copy code
"cloudsql_instances": {
        "anyOf": [
          {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          {
            "type": "null"
          }
        ],
        "title": "Cloud SQL Instances",
        "default": [
          "infrastructure-467509:europe-north2:primary"
        ],
        "examples": [
          "project:region:instance-id"
        ],
        "description": "List of Cloud SQL instance connection names to connect to. Format: {project}:{location}:{instance}"
      },
      "vpc_connector_name": {
        "anyOf": [
          {
            "type": "string"
          },
          {
            "type": "null"
          }
        ],
        "title": "VPC Connector Name",
        "description": "The name of the VPC connector to use for the Cloud Run job."
      },
      "service_account_name": {
        "anyOf": [
          {
            "type": "string"
          },
          {
            "type": "null"
          }
        ],
        "title": "Service Account Name",
        "default": "<mailto:prefect-worker-sa@infrastructure-467509.iam.gserviceaccount.com|prefect-worker-sa@infrastructure-467509.iam.gserviceaccount.com>",
        "examples": [
          "<mailto:service-account@example.iam.gserviceaccount.com|service-account@example.iam.gserviceaccount.com>"
        ],
        "description": "The name of the service account to use for the task execution of Cloud Run Job. By default Cloud Run jobs run as the default Compute Engine Service Account."
      },
      "prefect_api_key_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect API Key Secret",
        "examples": [
          {
            "secret": "prefect-api-key",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect API key. When provided, the secret will be used instead of the PREFECT_API_KEY environment variable."
      },
      "prefect_api_auth_string_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect API Auth String Secret",
        "default": {
          "secret": "prefect-api-auth",
          "version": "latest"
        },
        "examples": [
          {
            "secret": "prefect-api-auth-string",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect API auth string. When provided, the secret will be used instead of the PREFECT_API_AUTH_STRING environment variable."
      },
      "prefect_server_api_auth_string_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect Server API Auth String Secret",
        "default": {
          "secret": "prefect-api-auth",
          "version": "latest"
        },
        "examples": [
          {
            "secret": "prefect-server-api-auth-string",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect Server API auth string. When provided, the secret will be used instead of the PREFECT_SERVER_API_AUTH_STRING environment variable."
      }
    },
    "definitions": {
      "GcpCredentials": {
        "type": "object",
        "title": "GcpCredentials",
        "properties": {
          "project": {
            "anyOf": [
              {
                "type": "string"
              },
              {
                "type": "null"
              }
            ],
            "title": "Project",
            "default": null,
            "description": "The GCP project to use for the client."
          },
          "service_account_file": {
            "anyOf": [
              {
                "type": "string",
                "format": "path"
              },
              {
                "type": "null"
              }
            ],
            "title": "Service Account File",
            "default": null,
            "description": "Path to the service account JSON keyfile."
          },
          "service_account_info": {
            "anyOf": [
              {
                "type": "object",
                "additionalProperties": true
              },
              {
                "type": "null"
              }
            ],
            "title": "Service Account Info",
            "default": null,
            "description": "The contents of the keyfile as a dict."
          }
        },
        "description": "Block used to manage authentication with GCP. Google authentication is\nhandled via the `google.oauth2` module or through the CLI.\nSpecify either one of service `account_file` or `service_account_info`; if both\nare not specified, the client will try to detect the credentials following Google's\n[Application Default Credentials](<https://cloud.google.com/docs/authentication/application-default-credentials>).\nSee Google's [Authentication documentation](<https://cloud.google.com/docs/authentication#service-accounts>)\nfor details on inference and recommended authentication patterns.",
        "secret_fields": [
          "service_account_info.*"
        ],
        "block_type_slug": "gcp-credentials",
        "block_schema_references": {}
      },
      "SecretKeySelector": {
        "type": "object",
        "title": "SecretKeySelector",
        "required": [
          "secret",
          "version"
        ],
        "properties": {
          "secret": {
            "type": "string",
            "title": "Secret"
          },
          "version": {
            "type": "string",
            "title": "Version"
          }
        },
        "description": "SecretKeySelector is a data model for specifying a GCP secret to inject\ninto a Cloud Run V2 Job as an environment variable.\nFollows Cloud Run V2 rest API, docs:\n<https://cloud.google.com/run/docs/reference/rest/v2/Container#SecretKeySelector>"
      }
    },
    "description": "Default variables for the v2 Cloud Run worker.\n\nThe schema for this class is used to populate the `variables` section of the\ndefault base job template."
  },
  "job_configuration": {
    "env": "{{ env }}",
    "name": "{{ name }}",
    "labels": "{{ labels }}",
    "region": "{{ region }}",
    "command": "{{ command }}",
    "timeout": "{{ timeout }}",
    "job_body": {
      "client": "prefect",
      "template": {
        "template": {
          "timeout": "{{ timeout }}",
          "vpcAccess": {
            "connector": "{{ vpc_connector_name }}"
          },
          "containers": [
            {
              "env": [],
              "args": "{{ args }}",
              "image": "{{ image }}",
              "command": "{{ command }}",
              "resources": {
                "limits": {
                  "cpu": "{{ cpu }}",
                  "memory": "{{ memory }}"
                }
              }
            }
          ],
          "maxRetries": "{{ max_retries }}",
          "serviceAccount": "{{ service_account_name }}"
        }
      },
      "launchStage": "{{ launch_stage }}"
    },
    "keep_job": "{{ keep_job }}",
    "credentials": "{{ credentials }}",
    "env_from_secrets": "{{ env_from_secrets }}",
    "cloudsql_instances": "{{ cloudsql_instances }}",
    "prefect_api_key_secret": "{{ prefect_api_key_secret }}",
    "prefect_api_auth_string_secret": "{{ prefect_api_auth_string_secret }}",
    "prefect_server_api_auth_string_secret": "{{ prefect_server_api_auth_string_secret }}"
  }
}
First part of the config is left out:
m
Your message was too long, here's your message at the allowed limit: - I have the below config for a google cloud run worker and I am having issues with the prefect_server_api_auth_string_secret not being passed properly into the instance that is created when a run is triggered It works fine for the prefect_api_auth_string_secret - please help me figure out what is wrong with the config ``` cloudsql_instances { anyOf [ { type array items { type string } } { type null } ] title Cloud SQL Instances default [ infrastructure-467509 europe-north2 primary ] examples [ project region instance-id ] description List of Cloud SQL instance connection names to connect to Format {project} {location} {instance} } vpc_connector_name { anyOf [ { type string } { type null } ] title VPC Connector Name description The name of the VPC connector to use for the Cloud Run job } service_account_name { anyOf [ { type string } { type null } ] title Service Account Name default <mailto prefect-worker-sa@infrastructure-467509 iam gserviceaccount com|prefect-worker-sa@infrastructure-467509 iam gserviceaccount com> examples [ <mailto service-account@example iam gserviceaccount com|service-account@example iam gserviceaccount com> ] description The name of the service account to use for the task execution of Cloud Run Job By default Cloud Run jobs run as the default Compute Engine Service Account } prefect_api_key_secret { anyOf [ { $ref #/definitions/SecretKeySelector } { type null } ] title Prefect API Key Secret examples [ { secret prefect-api-key version latest } ] description The GCP secret to use for the Prefect API key When provided the secret will be used instead of the PREFECT_API_KEY environment variable } prefect_api_auth_string_secret { anyOf [ { $ref #/definitions/SecretKeySelector } { type null } ] title Prefect API Auth String Secret default { secret prefect-api-auth version latest } examples [ { secret prefect-api-auth-string version latest } ] description The GCP secret to use for the Prefect API auth string When provided the secret will be used instead of the PREFECT_API_AUTH_STRING environment variable } prefect_server_api_auth_string_secret { anyOf [ { $ref #/definitions/SecretKeySelector } { type null } ] title Prefect Server API Auth String Secret default { secret prefect-api-auth version latest } examples [ { secret prefect-server-api-auth-string version latest } ] description The GCP secret to use for the Prefect Server API auth string When provided the secret will be used instead of the PREFECT_SERVER_API_AUTH_STRING environment variable } } definitions { GcpCredentials { type object title GcpCredentials properties { project { anyOf [ { type string } { type null } ] title Project default null description The GCP project to use for the client } service_account_file { anyOf [ { type string format path } { type null } ] title Service Account File default null description Path to the service account JSON keyfile } service_account_info { anyOf [ { type object additionalProperties true } { type null } ] title Service Account Info default null description The contents of the keyfile as a dict } } description Block used to manage authentication with GCP Google authentication is\nhandled via the 
google oauth2
module or through the CLI
r
@Marvin - I have the below config for a google cloud run worker, and I am having issues with the "prefect_server_api_auth_string_secret" not being passed properly into the instance that is created when a run is triggered. It works fine for the "prefect_api_auth_string_secret" - please help me figure out what is wrong with the config:
Copy code
"cloudsql_instances": {
        "anyOf": [
          {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          {
            "type": "null"
          }
        ],
        "title": "Cloud SQL Instances",
        "default": [
          "infrastructure-467509:europe-north2:primary"
        ],
        "examples": [
          "project:region:instance-id"
        ],
        "description": "List of Cloud SQL instance connection names to connect to. Format: {project}:{location}:{instance}"
      },
      "vpc_connector_name": {
        "anyOf": [
          {
            "type": "string"
          },
          {
            "type": "null"
          }
        ],
        "title": "VPC Connector Name",
        "description": "The name of the VPC connector to use for the Cloud Run job."
      },
      "service_account_name": {
        "anyOf": [
          {
            "type": "string"
          },
          {
            "type": "null"
          }
        ],
        "title": "Service Account Name",
        "default": "<mailto:prefect-worker-sa@infrastructure-467509.iam.gserviceaccount.com|prefect-worker-sa@infrastructure-467509.iam.gserviceaccount.com>",
        "examples": [
          "<mailto:service-account@example.iam.gserviceaccount.com|service-account@example.iam.gserviceaccount.com>"
        ],
        "description": "The name of the service account to use for the task execution of Cloud Run Job. By default Cloud Run jobs run as the default Compute Engine Service Account."
      },
      "prefect_api_key_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect API Key Secret",
        "examples": [
          {
            "secret": "prefect-api-key",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect API key. When provided, the secret will be used instead of the PREFECT_API_KEY environment variable."
      },
      "prefect_api_auth_string_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect API Auth String Secret",
        "default": {
          "secret": "prefect-api-auth",
          "version": "latest"
        },
        "examples": [
          {
            "secret": "prefect-api-auth-string",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect API auth string. When provided, the secret will be used instead of the PREFECT_API_AUTH_STRING environment variable."
      },
      "prefect_server_api_auth_string_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect Server API Auth String Secret",
        "default": {
          "secret": "prefect-api-auth",
          "version": "latest"
        },
        "examples": [
          {
            "secret": "prefect-server-api-auth-string",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect Server API auth string. When provided, the secret will be used instead of the PREFECT_SERVER_API_AUTH_STRING environment variable."
      }
    },
    "definitions": {
      
            "title": "Project",
            "default": null,
            "description": "The GCP project to use for the client."
          },
          "service_account_file": {
            "anyOf": [
              {
                "type": "string",
                "format": "path"
              },
              {
                "type": "null"
              }
            ],
            "title": "Service Account File",
            "default": null,
            "description": "Path to the service account JSON keyfile."
          },
        
  },
  "job_configuration": {
    "env": "{{ env }}",
    "name": "{{ name }}",
    "labels": "{{ labels }}",
    "region": "{{ region }}",
    "command": "{{ command }}",
    "timeout": "{{ timeout }}",
    "job_body": {
      "client": "prefect",
      "template": {
        "template": {
          "timeout": "{{ timeout }}",
          "vpcAccess": {
            "connector": "{{ vpc_connector_name }}"
          },
          "containers": [
            {
              "env": [],
              "args": "{{ args }}",
              "image": "{{ image }}",
              "command": "{{ command }}",
              "resources": {
                "limits": {
                  "cpu": "{{ cpu }}",
                  "memory": "{{ memory }}"
                }
              }
            }
          ],
          "maxRetries": "{{ max_retries }}",
          "serviceAccount": "{{ service_account_name }}"
        }
      },
      "launchStage": "{{ launch_stage }}"
    },
    "keep_job": "{{ keep_job }}",
    "credentials": "{{ credentials }}",
    "env_from_secrets": "{{ env_from_secrets }}",
    "cloudsql_instances": "{{ cloudsql_instances }}",
    "prefect_api_key_secret": "{{ prefect_api_key_secret }}",
    "prefect_api_auth_string_secret": "{{ prefect_api_auth_string_secret }}",
    "prefect_server_api_auth_string_secret": "{{ prefect_server_api_auth_string_secret }}"
  }
}
(Some parts are left out)
m
Your message was too long, here's your message at the allowed limit: - I have the below config for a google cloud run worker and I am having issues with the prefect_server_api_auth_string_secret not being passed properly into the instance that is created when a run is triggered It works fine for the prefect_api_auth_string_secret - please help me figure out what is wrong with the config ``` cloudsql_instances { anyOf [ { type array items { type string } } { type null } ] title Cloud SQL Instances default [ infrastructure-467509 europe-north2 primary ] examples [ project region instance-id ] description List of Cloud SQL instance connection names to connect to Format {project} {location} {instance} } vpc_connector_name { anyOf [ { type string } { type null } ] title VPC Connector Name description The name of the VPC connector to use for the Cloud Run job } service_account_name { anyOf [ { type string } { type null } ] title Service Account Name default <mailto prefect-worker-sa@infrastructure-467509 iam gserviceaccount com|prefect-worker-sa@infrastructure-467509 iam gserviceaccount com> examples [ <mailto service-account@example iam gserviceaccount com|service-account@example iam gserviceaccount com> ] description The name of the service account to use for the task execution of Cloud Run Job By default Cloud Run jobs run as the default Compute Engine Service Account } prefect_api_key_secret { anyOf [ { $ref #/definitions/SecretKeySelector } { type null } ] title Prefect API Key Secret examples [ { secret prefect-api-key version latest } ] description The GCP secret to use for the Prefect API key When provided the secret will be used instead of the PREFECT_API_KEY environment variable } prefect_api_auth_string_secret { anyOf [ { $ref #/definitions/SecretKeySelector } { type null } ] title Prefect API Auth String Secret default { secret prefect-api-auth version latest } examples [ { secret prefect-api-auth-string version latest } ] description The GCP secret to use for the Prefect API auth string When provided the secret will be used instead of the PREFECT_API_AUTH_STRING environment variable } prefect_server_api_auth_string_secret { anyOf [ { $ref #/definitions/SecretKeySelector } { type null } ] title Prefect Server API Auth String Secret default { secret prefect-api-auth version latest } examples [ { secret prefect-server-api-auth-string version latest } ] description The GCP secret to use for the Prefect Server API auth string When provided the secret will be used instead of the PREFECT_SERVER_API_AUTH_STRING environment variable } } definitions { title Project default null description The GCP project to use for the client } service_account_file { anyOf [ { type string format path } { type null } ] title Service Account File default null description Path to the service account JSON keyfile } } job_configuration { env {{ env }} name {{ name }} labels {{ labels }} region {{ region }} command {{ command }} timeout {{ timeout }} job_body { client prefect template { template { timeout {{ timeout }} vpcAccess { connector {{ vpc_connector_name }} } containers [ { env [] args {{ args }} image {{ image }} command {{ command }} resources { limits { cpu {{ cpu }} memory {{
r
@Marvin - I have some variables from my work pool (a google cloud run work pool), and some of the variable i set in the work pool configuration are not being passed into the job runs - why is this?
m
✅ thought for 47.1 seconds
To help diagnose your issue, you can use:
Copy code
bash
prefect work-pool inspect <your-work-pool-name>
This will show you the current configuration of your work pool, including variables. To properly configure variables that will be passed to your job runs, you have several options: 1. Work Pool Configuration
Copy code
python
   from prefect import deploy
   
   deployment = flow.from_source(
       source="<https://github.com/your/repo>",
       entrypoint="path/to/flow.py:flow_func",
   ).deploy(
       name="my-deployment",
       work_pool_name="my-cloud-run-pool",
       job_variables={
           "your_variable": "value",
           # other variables here
       }
   )
2. Update Existing Work Pool Variables You can update work pool variables using the UI or CLI:
Copy code
bash
   prefect work-pool update my-cloud-run-pool --job-variables='{"key": "value"}'
To help you further, could you please share: 1. The specific variables you're trying to set 2. How you're setting them (UI, CLI, or code) 3. Where in your flow you're trying to access these variables This will help me provide more targeted guidance for your specific use case.
r
@Marvin I would like to pass a ""SecretKeySelector" - I have one which is working, but another which is not
m
✅ thought for 27.0 seconds
To help you troubleshoot your SecretKeySelector configuration, I'll need a few details: 1. Could you share (without revealing sensitive values): - The configuration of the SecretKeySelector that's working - The configuration of the one that's not working 2. How are you defining these SecretKeySelectors? Are you using: - The work pool configuration in the UI - A deployment configuration file - The Python API Based on recent updates to Prefect, there are specific ways to configure secrets for Cloud Run jobs. For example, if you're trying to set environment variables from GCP secrets, the structure should look like this:
Copy code
json
"env_from_secrets": {
  "ENV_VAR_NAME": {
    "secret": "SECRET_NAME",
    "version": "latest"
  }
}
If you can share your configurations (with sensitive values redacted), I can help identify why one is working and the other isn't. Common issues include: 1. Incorrect secret name or path 2. Missing permissions to access the secret 3. Incorrect configuration structure 4. Version specification issues Could you please provide: 1. The structure of both SecretKeySelector configurations (working and non-working) 2. How you're defining these in your work pool or deployment 3. Any error messages you're seeing (if any) This will help me provide more specific guidance for your situation.
r
@Marvin - I am trying to pass these two:
Copy code
"prefect_api_auth_string_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect API Auth String Secret",
        "default": {
          "secret": "prefect-api-auth",
          "version": "latest"
        },
        "examples": [
          {
            "secret": "prefect-api-auth-string",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect API auth string. When provided, the secret will be used instead of the PREFECT_API_AUTH_STRING environment variable."
      },
      "prefect_server_api_auth_string_secret": {
        "anyOf": [
          {
            "$ref": "#/definitions/SecretKeySelector"
          },
          {
            "type": "null"
          }
        ],
        "title": "Prefect Server API Auth String Secret",
        "examples": [
          {
            "secret": "prefect-server-api-auth-string",
            "version": "latest"
          }
        ],
        "description": "The GCP secret to use for the Prefect Server API auth string. When provided, the secret will be used instead of the PREFECT_SERVER_API_AUTH_STRING environment variable."
      }
The "prefect-api-auth-string" is working as intended, but "prefect-server-api-auth-string" is not
m
✅ thought for 13.5 seconds
Looking at your configuration, I notice a few key things: 1. The working configuration (
prefect_api_auth_string_secret
): - Has a default value specified - References secret "prefect-api-auth" - Has "latest" version specified 2. The non-working configuration (
prefect_server_api_auth_string_secret
): - Does not have a default value - References secret "prefect-server-api-auth-string" - Has "latest" version specified in examples but not in a default To help diagnose this, I need to know: 1. In Google Cloud Secret Manager, can you confirm: - Does the secret "prefect-server-api-auth-string" exist? - Does the service account running your Cloud Run jobs have access to both secrets? 2. How are you setting these values in your work pool configuration? Are you using: - The UI? - A JSON configuration file? - The Python API? 3. Are you seeing any specific error messages in your job logs? Here's what I recommend trying: 1. First, try adding a default value to the non-working secret (similar to the working one):
Copy code
json
"prefect_server_api_auth_string_secret": {
    "anyOf": [
      {
        "$ref": "#/definitions/SecretKeySelector"
      },
      {
        "type": "null"
      }
    ],
    "title": "Prefect Server API Auth String Secret",
    "default": {
      "secret": "prefect-server-api-auth-string",
      "version": "latest"
    },
    "examples": [
      {
        "secret": "prefect-server-api-auth-string",
        "version": "latest"
      }
    ],
    "description": "The GCP secret to use for the Prefect Server API auth string..."
}
2. Verify the secret access:
Copy code
bash
# Check if your service account can access the secret
gcloud secrets versions access "latest" --secret="prefect-server-api-auth-string"
3. Check your work pool configuration:
Copy code
bash
prefect work-pool inspect your-work-pool-name
Could you please: 1. Confirm if the secret exists in Google Cloud Secret Manager 2. Share how you're configuring these values in your work pool 3. Share any error messages you're seeing in your job logs This will help me provide more specific guidance for resolving the issue.
r
@Marvin I get this error now:
Copy code
returned "template.template.containers[0].env: Secret environment variable overlaps non secret environment variable: PREFECT_SERVER_API_AUTH_STRING". Details: "[{'@type': '<http://type.googleapis.com/google.rpc.BadRequest|type.googleapis.com/google.rpc.BadRequest>', 'fieldViolations': [{'field': 'template.template.containers[0].env', 'description': 'Secret environment variable overlaps non secret environment variable: PREFECT_SERVER_API_AUTH_STRING'}]}]">
It seems like the PREFECT_SERVER_API_AUTH_STRING is now being passed twice
m
✅ thought for 56.8 seconds
To fix this issue, you need to ensure that 
PREFECT_SERVER_API_AUTH_STRING
is only being set once. Here's how to troubleshoot and fix it: 1. Check Your Current Configuration - Look in your work pool configuration for any direct environment variable settings - Check your deployment configuration for any environment variable settings - Review your base job template for any hardcoded environment variables 2. Choose One Method You should use either: - Secret-based configuration (recommended for production) - Direct environment variable setting (not recommended for sensitive data) But not both at the same time. 3. Recommended Fix: 1. Remove any direct setting of 
PREFECT_SERVER_API_AUTH_STRING
from your environment variables 2. Keep only the secret-based configuration:
Copy code
json
   "prefect_server_api_auth_string_secret": {
       "secret": "prefect-server-api-auth-string",
       "version": "latest"
   }
To help you implement this fix, could you please: 1. Check your work pool configuration (you can use 
prefect work-pool inspect your-pool-name
) and verify if 
PREFECT_SERVER_API_AUTH_STRING
is set directly in the environment variables 2. Check your deployment configuration for any environment variable settings Once you share these details, I can provide more specific guidance on where to remove the duplicate environment variable setting. Remember: Cloud Run doesn't allow the same environment variable to be set both as a regular environment variable and as a secret-sourced variable, which is what's causing your error.
r
@Marvin - Okay that issue now seems to be fixed. I still have an issue though. I am also trying to pass a cloud sql instance variable through the work pool, however, i dont see the cloud sql instance in the environment variables for the yaml file of the cloud run job.
m
✅ thought for 57.6 seconds
Based on the research and your issue, I understand that you're trying to configure a Cloud SQL instance connection in your Cloud Run work pool, but it's not showing up in the YAML configuration. Here's what we need to check: 1. Job Template Configuration The Cloud SQL instance should be configured in the base job template for your Cloud Run work pool. This is typically done through the 
cloud_sql_connections
field in the template. 2. Correct Configuration Structure The configuration should look something like this in your work pool's job template:
Copy code
json
   {
     "job_configuration": {
       "cloud_sql_connections": ["your-project:your-region:your-instance"]
     }
   }
To troubleshoot this: 1. First, let's check your current work pool configuration:
Copy code
bash
prefect work-pool inspect your-work-pool-name
2. You can update the work pool configuration to include the Cloud SQL instance using either: - The UI: Navigate to your work pool and edit the job template - The CLI: Update the work pool with a new job template
Copy code
bash
   prefect work-pool create your-work-pool-name --type cloud-run --base-job-template path/to/template.json --overwrite
Could you please share: 1. The current job template configuration you're using 2. How you're trying to specify the Cloud SQL instance 3. The exact instance connection name you're trying to use This will help me provide more specific guidance on how to properly configure the Cloud SQL instance connection in your work pool. Also, make sure that: 1. The service account used by Cloud Run has the necessary permissions to access the Cloud SQL instance 2. The Cloud SQL instance connection name is in the correct format: 
project:region:instance
3. The Cloud SQL instance is in the same region as your Cloud Run service (or configured for cross-region access)
Open in Slack
PreviousNext
Powered by