https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I'm running prefect self-hosted serve...
# ask-marvin
u
@Marvin I'm running prefect self-hosted server on GCP Cloud Run service with Identity-Aware Proxy enabled. And, I want to deploy flow on GitHub Action. Authentication is done in GitHub Action, but prefect cli seem not to use the authentications. What should I Do?
m
✅ thought for 64.8 seconds
Based on the research and my understanding of GCP IAP authentication, here's what you need to do to make Prefect CLI work with IAP authentication in GitHub Actions: 1. First, you'll need to set up authentication for IAP in your GitHub Actions workflow. This typically involves: - Creating a service account with appropriate permissions - Creating and downloading a service account key - Setting up the key as a GitHub secret 2. Then, you'll need to modify your GitHub Actions workflow to handle IAP authentication. Here's a suggested approach:
Copy code
yaml
name: Deploy Prefect Flow

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      # Set up Python
      - uses: actions/setup-python@v4
        with:
          python-version: '3.10'
          
      # Authenticate to Google Cloud
      - id: 'auth'
        uses: 'google-github-actions/auth@v1'
        with:
          credentials_json: '${{ secrets.GCP_SA_KEY }}'
          
      # Get IAP authentication token
      - name: Get IAP token
        run: |
          # Install necessary tools
          pip install google-auth requests
          
          # Generate IAP token and store it
          python -c '
          from google.oauth2 import service_account
          from google.auth.transport.requests import Request
          import os
          
          credentials = service_account.Credentials.from_service_account_info(
              ${{ secrets.GCP_SA_KEY }},
              ["<https://www.googleapis.com/auth/cloud-platform>"]
          )
          
          # Force token refresh
          request = Request()
          credentials.refresh(request)
          
          # Set the token as an environment variable
          with open(os.environ["GITHUB_ENV"], "a") as f:
              f.write(f"IAP_TOKEN={credentials.token}\n")
          '
          
      # Set up Prefect
      - name: Setup Prefect
        run: |
          pip install prefect
          # Set your Prefect API URL
          prefect config set PREFECT_API_URL=${{ secrets.PREFECT_API_URL }}
          
      # Deploy your flow
      - name: Deploy Flow
        env:
          # Make the token available to the Prefect CLI
          PREFECT_API_HEADERS: '{"Authorization": "Bearer ${{ env.IAP_TOKEN }}"}'
        run: |
          prefect deploy ...  # Your deployment command here
Key points to note: 1. You'll need to store these secrets in your GitHub repository: - `GCP_SA_KEY`: Your service account key JSON - `PREFECT_API_URL`: Your Prefect server URL 2. The crucial part is setting the 
PREFECT_API_HEADERS
environment variable with the IAP token. This tells Prefect CLI to include the IAP authentication token in its requests. 3. Make sure your service account has the necessary IAP permissions. It typically needs: - 
roles/iap.httpsResourceAccessor
- Any additional roles needed for your specific Prefect server setup 4. The service account should be authorized to access the IAP-protected service (your Prefect server). If you're still experiencing issues, you can debug the authentication by: 1. Adding debug logging to your Python script that generates the token 2. Checking the Cloud Run logs for authentication failures 3. Testing the token manually with a curl command before running Prefect commands Would you like me to provide more details about any specific part of this solution?
u
@Marvin is the 
PREFECT_API_HEADERS
existing?
m
✅ thought for 47.5 seconds
I apologize for my previous response where I suggested using 
PREFECT_API_HEADERS
. That was incorrect as this configuration option does not exist in Prefect. For authenticating with IAP in Prefect, you'll need to use a different approach. Here are two possible solutions: 1. Use an HTTP Proxy:
Copy code
python
# Set these environment variables
PREFECT_API_URL="your-server-url"
HTTPS_PROXY="your-proxy-that-handles-IAP-auth"
2. Create a Custom HTTP Client: You can create a custom HTTP client that includes the IAP token in its requests. Here's how you might do this:
Copy code
python
from prefect.settings import PREFECT_API_URL
from prefect.client.schemas.actions import ClientAPI
from prefect.client.utilities import get_client
from httpx import Client

# Create a custom client with IAP authentication
def get_iap_client():
    credentials = service_account.Credentials.from_service_account_info(
        # your service account info here
        scopes=["<https://www.googleapis.com/auth/cloud-platform>"]
    )
    
    request = Request()
    credentials.refresh(request)
    
    headers = {
        "Authorization": f"Bearer {credentials.token}"
    }
    
    client = Client(
        base_url=PREFECT_API_URL.value(),
        headers=headers
    )
    return client

# Use the custom client with Prefect
api_client = ClientAPI(client=get_iap_client())
For GitHub Actions specifically, you might want to: 1. Set up authentication using the Google Cloud Auth action 2. Use the authenticated Google Cloud SDK to generate IAP tokens 3. Use those tokens with a custom HTTP client in your Prefect scripts Would you like me to provide more detailed examples of either of these approaches? Also, it would be helpful to know which specific Prefect commands you're trying to run in your GitHub Actions workflow.
u
@Marvin what about 
PREFECT_CLIENT_CUSTOM_HEADERS
? check this document
m
✅ thought for 48.4 seconds
You're absolutely right! I apologize for my oversight. The link you provided points to the correct solution. The 
PREFECT_CLIENT_CUSTOM_HEADERS
setting is indeed the proper way to add custom headers to Prefect API requests. For your GitHub Actions workflow with IAP authentication, you can modify your workflow to use this setting:
Copy code
yaml
name: Deploy Prefect Flow

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      # Set up Python
      - uses: actions/setup-python@v4
        with:
          python-version: '3.10'
          
      # Authenticate to Google Cloud
      - id: 'auth'
        uses: 'google-github-actions/auth@v1'
        with:
          credentials_json: '${{ secrets.GCP_SA_KEY }}'
          
      # Get IAP token
      - name: Get IAP token
        run: |
          pip install google-auth requests
          
          python -c '
          from google.oauth2 import service_account
          from google.auth.transport.requests import Request
          import os
          import json
          
          credentials = service_account.Credentials.from_service_account_info(
              json.loads("""${{ secrets.GCP_SA_KEY }}"""),
              ["<https://www.googleapis.com/auth/cloud-platform>"]
          )
          
          request = Request()
          credentials.refresh(request)
          
          # Set the headers as a JSON string
          headers = json.dumps({"Authorization": f"Bearer {credentials.token}"})
          with open(os.environ["GITHUB_ENV"], "a") as f:
              f.write(f"PREFECT_CLIENT_CUSTOM_HEADERS={headers}\n")
          '
          
      # Set up Prefect
      - name: Setup Prefect
        run: |
          pip install prefect
          prefect config set PREFECT_API_URL=${{ secrets.PREFECT_API_URL }}
          
      # Deploy your flow
      - name: Deploy Flow
        run: |
          prefect deploy ...  # Your deployment command here
Key differences from my previous incorrect response: 1. Uses 
PREFECT_CLIENT_CUSTOM_HEADERS
instead of the non-existent 
PREFECT_API_HEADERS
2. The headers need to be provided as a JSON string 3. This setting will be automatically picked up by all Prefect client operations This is a much cleaner solution as it uses the officially supported method for adding custom headers to Prefect API requests. The setting will be used for all API calls made by the Prefect client, ensuring your IAP authentication is properly handled. Would you like me to explain any part of this solution in more detail?
u
@Marvin Actually, I've already tried that. the response of 
prefect deploy ~
require IAP login.
m
✅ thought for 33.2 seconds
If you're still getting an IAP login prompt even after setting 
PREFECT_CLIENT_CUSTOM_HEADERS
, this suggests that either: 1. The IAP token isn't being properly generated/set, or 2. The token isn't being properly formatted in the headers Let's try to debug this. Could you: 1. First, verify that the IAP token is being generated correctly by adding some debug output:
Copy code
yaml
- name: Get IAP token and debug
  run: |
    python -c '
    from google.oauth2 import service_account
    from google.auth.transport.requests import Request
    import os
    import json

    credentials = service_account.Credentials.from_service_account_info(
        json.loads("""${{ secrets.GCP_SA_KEY }}"""),
        ["<https://www.googleapis.com/auth/cloud-platform>"]
    )

    request = Request()
    credentials.refresh(request)
    
    # Debug print the token (careful with sensitive info in logs)
    print("Token generated successfully:", bool(credentials.token))
    print("Token length:", len(credentials.token) if credentials.token else 0)
    
    # You might also want to verify the token format
    headers = json.dumps({"Authorization": f"Bearer {credentials.token}"})
    print("Headers JSON format:", headers)
    
    with open(os.environ["GITHUB_ENV"], "a") as f:
        f.write(f"PREFECT_CLIENT_CUSTOM_HEADERS={headers}\n")
    '
2. Could you also try making a test request to your IAP-protected endpoint using curl with the same token to verify it works:
Copy code
yaml
- name: Test IAP token
  run: |
    TOKEN=$(python -c '
    # ... same token generation code as above ...
    print(credentials.token)
    ')
    
    curl -v -H "Authorization: Bearer $TOKEN" ${{ secrets.PREFECT_API_URL }}/api/health
3. Additionally, you might want to try setting the audience explicitly when generating the token:
Copy code
python
credentials = service_account.Credentials.from_service_account_info(
    json.loads("""${{ secrets.GCP_SA_KEY }}"""),
    ["<https://www.googleapis.com/auth/cloud-platform>"],
    target_audience=f"<https://iap.googleapis.com/projects/{PROJECT_NUMBER}/global/backend-services/{BACKEND_SERVICE_ID}>"
)
Could you: 1. Confirm if you see the token being generated successfully in the GitHub Actions logs? 2. Share if you're able to make a successful curl request to your endpoint using the same token? 3. Let me know if you're using any specific IAP audience configuration? This will help us narrow down where exactly the authentication is failing.
✅ thought for 44.7 seconds
Based on the research and error message, here are a few potential solutions to try: 1. First, check if PREFECT_API_KEY is set:
Copy code
bash
# In your GitHub Actions workflow, verify no API key is set
echo "Checking for PREFECT_API_KEY..."
env | grep PREFECT_API_KEY
If it exists, remove it as it might interfere with the authentication. 2. Set up Basic Authentication: If your self-hosted Prefect server uses basic authentication, you need to configure it in your GitHub Actions:
Copy code
yaml
- name: Setup Prefect
  run: |
    pip install prefect
    prefect config set PREFECT_API_URL=${{ secrets.PREFECT_API_URL }}
    # If using basic auth, set the credentials
    prefect config set PREFECT_API_AUTH_CREDENTIALS=${{ secrets.PREFECT_API_AUTH_CREDENTIALS }}  # format: "username:password"
3. Verify Server Configuration: On your Prefect server, ensure you have the correct authentication settings. If using basic auth, your server configuration should include:
Copy code
python
server:
  api:
    auth_string: "admin:password"  # Replace with your actual credentials
4. Check Headers: Let's verify the headers being sent with your requests:
Copy code
yaml
- name: Debug Headers
  run: |
    python -c '
    import os
    import json
    
    # Print current headers
    headers = os.getenv("PREFECT_CLIENT_CUSTOM_HEADERS", "{}")
    print("Current headers:", headers)
    
    # For basic auth + IAP, you might need to combine both
    combined_headers = {
        "Authorization": "Bearer YOUR_IAP_TOKEN",
        # Add basic auth if needed
        # "Authorization-Basic": "base64_encoded_credentials"
    }
    
    # Set the combined headers
    with open(os.environ["GITHUB_ENV"], "a") as f:
        f.write(f"PREFECT_CLIENT_CUSTOM_HEADERS={json.dumps(combined_headers)}\n")
    '
Could you please: 1. Confirm if your Prefect server is using any authentication method besides IAP (like basic auth)? 2. Check if 
PREFECT_API_KEY
is set in your environment? 3. Try running a simple Prefect command with debug logging enabled:
Copy code
bash
PREFECT_LOGGING_LEVEL=DEBUG prefect work-pool ls
This will show more detailed information about the request and response. The error suggests that while IAP authentication succeeded, there might be an additional layer of authentication needed for the Prefect server itself. Let me know which of these you'd like to try first or if you need more details about any of these approaches.
u
@Marvin I'm running self-hosted prefect server. Do I need to use api key?
m
✅ thought for 41.2 seconds
No, you do not need (and should not use) an API key for a self-hosted Prefect server. In fact, having 
PREFECT_API_KEY
set can cause authentication issues with a self-hosted server. Here's what you should do: 1. Remove API Key if Set:
Copy code
bash
# Check if API key is set
unset PREFECT_API_KEY
# Also remove it from any environment variables in GitHub Actions if present
2. Use Basic Authentication Instead: For a self-hosted server, you should use basic authentication. This involves: On the server side:
Copy code
python
# In your server configuration
server:
  api:
    auth_string: "admin:yourpassword"  # Replace with your desired credentials
On the client side (GitHub Actions):
Copy code
yaml
- name: Setup Prefect
  run: |
    prefect config set PREFECT_API_URL=${{ secrets.PREFECT_API_URL }}
    prefect config set PREFECT_API_AUTH_CREDENTIALS=${{ secrets.PREFECT_API_AUTH_CREDENTIALS }}  # format: "username:password"
3. Combine with IAP: Since you're using IAP, you'll need both the IAP token and basic auth. Your headers should look like:
Copy code
yaml
- name: Set Headers
  run: |
    python -c '
    import json
    import os
    
    # Your existing IAP token generation code here
    # ...
    
    headers = {
        "Authorization": f"Bearer {credentials.token}"
    }
    
    with open(os.environ["GITHUB_ENV"], "a") as f:
        f.write(f"PREFECT_CLIENT_CUSTOM_HEADERS={json.dumps(headers)}\n")
    '
The key points are: 1. Do not use 
PREFECT_API_KEY
with self-hosted server 2. Use basic authentication if you need authentication 3. Combine basic auth with your IAP token for the complete authentication chain Would you like me to provide more details about setting up basic authentication for your self-hosted server, or would you prefer to try removing the API key first to see if that resolves the issue?
u
@Marvin i got these error below, Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.13.5/x64/lib/python3.13/site-packages/prefect/cli/_utilities.py", line 44, in wrapper return fn(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.13.5/x64/lib/python3.13/site-packages/prefect/cli/_types.py", line 156, in sync_fn return asyncio.run(async_fn(*args, **kwargs)) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.13.5/x64/lib/python3.13/asyncio/runners.py", line 195, in run return runner.run(main) ~~~~~~~~~~^^^^^^ File "/opt/hostedtoolcache/Python/3.13.5/x64/lib/python3.13/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/opt/hostedtoolcache/Python/3.13.5/x64/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete return future.result() ~~~~~~~~~~~~~^^ File "/opt/hostedtoolcache/Python/3.13.5/x64/lib/python3.13/site-packages/prefect/cli/work_pool.py", line 295, in ls pools = await client.read_work_pools() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.13.5/x64/lib/python3.13/site-packages/prefect/client/orchestration/_work_pools/client.py", line 454, in read_work_pools response = await self.request("POST", "/work_pools/filter", json=body) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.13.5/x64/lib/python3.13/site-packages/prefect/client/orchestration/base.py", line 53, in request return await self._client.send(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/hostedtoolcache/Python/3.13.5/x64/lib/python3.13/site-packages/prefect/client/base.py", line 379, in send response.raise_for_status() ~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/opt/hostedtoolcache/Python/3.13.5/x64/lib/python3.13/site-packages/prefect/client/base.py", line 163, in raise_for_status raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.cause prefect.exceptions.PrefectHTTPStatusError: Client error '401 Unauthorized' for url 'https://prefect-server-1030217702417.asia-northeast3.run.app/api/work_pools/filter'
m
✅ thought for 77.5 seconds
Based on the error and the context, let's try to debug this step by step: 1. First, let's verify your current IAP token setup. Could you try this modified script that includes more debugging information:
Copy code
yaml
- name: Debug Authentication
  run: |
    python -c '
    from google.oauth2 import service_account
    from google.auth.transport.requests import Request
    import os
    import json
    import requests

    # Load credentials
    credentials = service_account.Credentials.from_service_account_info(
        json.loads("""${{ secrets.GCP_SA_KEY }}"""),
        ["<https://www.googleapis.com/auth/cloud-platform>"]
    )

    request = Request()
    credentials.refresh(request)

    # Print token info (safely)
    print("Token exists:", bool(credentials.token))
    print("Token length:", len(credentials.token) if credentials.token else 0)

    # Test the token with a direct request
    headers = {"Authorization": f"Bearer {credentials.token}"}
    test_url = "${{ secrets.PREFECT_API_URL }}/api/health"
    print("\nTesting direct request to health endpoint...")
    response = requests.get(test_url, headers=headers)
    print(f"Status code: {response.status_code}")
    
    # Set headers for Prefect
    headers_json = json.dumps(headers)
    print("\nSetting headers:", headers_json)
    with open(os.environ["GITHUB_ENV"], "a") as f:
        f.write(f"PREFECT_CLIENT_CUSTOM_HEADERS={headers_json}\n")
    '
2. Then, let's verify the Prefect configuration:
Copy code
yaml
- name: Debug Prefect Config
  run: |
    echo "Current Prefect configuration:"
    prefect config view
    
    # Try a simple API request with debug logging
    PREFECT_LOGGING_LEVEL=DEBUG prefect work-pool ls
3. Also, could you verify your server configuration? On your self-hosted Prefect server, check:
Copy code
python
# prefect server configuration
server:
  api:
    # Check if any authentication is configured here
    auth_string: null  # or "username:password" if using basic auth
4. Try setting these environment variables explicitly:
Copy code
yaml
- name: Setup Prefect with Debug
  env:
    PREFECT_LOGGING_LEVEL: DEBUG
    PREFECT_CLIENT_CONNECT_TIMEOUT: 60  # Increase timeout for debugging
  run: |
    # Clear any existing API key
    unset PREFECT_API_KEY
    
    # Set the API URL
    prefect config set PREFECT_API_URL=${{ secrets.PREFECT_API_URL }}
    
    # Try the command
    prefect work-pool ls
Could you please: 1. Run these debug steps and share the output (excluding sensitive information)? 2. Confirm if your Prefect server is using any authentication method besides IAP? 3. Can you access the Prefect UI through your browser when using IAP authentication? This will help us understand: - If the IAP token is being generated correctly - If the token is being properly passed to Prefect - If there are any additional authentication layers we need to handle - If there might be an issue with the token scope or permissions
u
@Marvin below is settings of my prefect self-hosted server. Is the base-auth disabled now? api: { "url": "***", "auth_string": null, "key": null, "tls_insecure_skip_verify": false, "ssl_cert_file": null, "enable_http2": false, "request_timeout": 60 }
m
✅ thought for 12.3 seconds
Yes, looking at your server configuration:
Copy code
json
api: {
  "auth_string": null,  // Basic auth is disabled
  "key": null,         // API key auth is disabled
  ...
}
Both basic authentication (
auth_string: null
) and API key authentication (
key: null
) are disabled on your server. This means: 1. The server is relying solely on IAP for authentication 2. No additional Prefect-level authentication is enabled 3. You should NOT set: - 
PREFECT_API_AUTH_CREDENTIALS
- 
PREFECT_API_KEY
For your GitHub Actions workflow, you only need to handle the IAP authentication. Let's modify your setup to be cleaner:
Copy code
yaml
- name: Setup IAP Authentication
  run: |
    python -c '
    from google.oauth2 import service_account
    from google.auth.transport.requests import Request
    import os
    import json
    
    # Generate IAP token
    credentials = service_account.Credentials.from_service_account_info(
        json.loads("""${{ secrets.GCP_SA_KEY }}"""),
        ["<https://www.googleapis.com/auth/cloud-platform>"]
    )
    
    request = Request()
    credentials.refresh(request)
    
    # Set only the IAP header
    headers = {
        "Authorization": f"Bearer {credentials.token}"
    }
    
    # Set for Prefect client
    with open(os.environ["GITHUB_ENV"], "a") as f:
        f.write(f"PREFECT_CLIENT_CUSTOM_HEADERS={json.dumps(headers)}\n")
    '

- name: Setup Prefect
  run: |
    # Only set the API URL
    prefect config set PREFECT_API_URL=${{ secrets.PREFECT_API_URL }}

- name: Deploy Flow
  run: |
    prefect deploy ...  # Your deployment command
Since your server is configured to use only IAP authentication, we should focus on ensuring the IAP token is correct and properly formatted. Could you try this simplified setup and let me know if you still get the 401 error? Also, to verify the IAP token is working correctly, you could try a test request:
Copy code
yaml
- name: Test IAP Authentication
  run: |
    python -c '
    import requests
    import os
    import json
    
    headers = json.loads(os.environ["PREFECT_CLIENT_CUSTOM_HEADERS"])
    response = requests.get(
        "${{ secrets.PREFECT_API_URL }}/api/health",
        headers=headers
    )
    print(f"Status code: {response.status_code}")
    print(f"Response: {response.text if response.ok else "Failed"}")
    '
This will help verify if the IAP token is working correctly before attempting Prefect commands.
u
@Marvin here's the test result Run python -c ' python -c ' import requests import os import json headers = json.loads(os.environ["PREFECT_CLIENT_CUSTOM_HEADERS"]) response = requests.get( "https://prefect-server-1030217702417.asia-northeast3.run.app/api/api/health", headers=headers ) print(f"Status code: {response.status_code}") print(f"Response: {response.text if response.ok else "Failed"}") ' shell: /usr/bin/bash -e {0} env: PREFECT_API_URL: https://prefect-server-1030217702417.asia-northeast3.run.app/api PREFECT_CLIENT_CSRF_SUPPORT_ENABLED: false pythonLocation: /opt/hostedtoolcache/Python/3.13.5/x64 PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.13.5/x64/lib/pkgconfig Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.13.5/x64 Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.13.5/x64 Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.13.5/x64 LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.13.5/x64/lib CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /home/runner/work/ml-data-pipeline/ml-data-pipeline/gha-creds-eef47a1064facdaa.json GOOGLE_APPLICATION_CREDENTIALS: /home/runner/work/ml-data-pipeline/ml-data-pipeline/gha-creds-eef47a1064facdaa.json GOOGLE_GHA_CREDS_PATH: /home/runner/work/ml-data-pipeline/ml-data-pipeline/gha-creds-eef47a1064facdaa.json CLOUDSDK_CORE_PROJECT: solvook-infra CLOUDSDK_PROJECT: solvook-infra GCLOUD_PROJECT: solvook-infra GCP_PROJECT: solvook-infra GOOGLE_CLOUD_PROJECT: solvook-infra PREFECT_CLIENT_CUSTOM_HEADERS: {"Authorization":"***"} Status code: 401 Response: Failed
m
✅ thought for 15.6 seconds
I notice an issue with your test URL. You have a duplicate "api" in the path:
Copy code
python
"<https://prefect-server-1030217702417.asia-northeast3.run.app/api/api/health>"
                                                           ^^^^^^^^
The correct URL should be:
Copy code
python
"<https://prefect-server-1030217702417.asia-northeast3.run.app/api/health>"
Let's modify the test script to use the correct URL and add more debugging information:
Copy code
yaml
- name: Test IAP Authentication
  run: |
    python -c '
    import requests
    import os
    import json
    
    # Get the base URL from environment
    base_url = os.environ["PREFECT_API_URL"]
    
    # Remove any trailing /api if present
    if base_url.endswith("/api"):
        base_url = base_url[:-4]
    
    # Construct the health check URL
    health_url = f"{base_url}/api/health"
    
    # Get headers
    headers = json.loads(os.environ["PREFECT_CLIENT_CUSTOM_HEADERS"])
    
    print(f"Testing URL: {health_url}")
    print(f"Headers (safely): {json.dumps({k: "***" if k.lower() == "authorization" else v for k, v in headers.items()})}")
    
    # Make request with debug info
    try:
        response = requests.get(
            health_url,
            headers=headers,
            timeout=30
        )
        print(f"\nStatus code: {response.status_code}")
        print(f"Response headers: {dict(response.headers)}")
        print(f"Response: {response.text if response.ok else "Failed"}")
    except Exception as e:
        print(f"Request failed with error: {str(e)}")
    '
Also, let's verify the IAP token generation process:
Copy code
yaml
- name: Debug IAP Token Generation
  run: |
    python -c '
    from google.oauth2 import service_account
    from google.auth.transport.requests import Request
    import os
    import json
    
    # Load credentials
    credentials = service_account.Credentials.from_service_account_info(
        json.loads(open(os.environ["GOOGLE_APPLICATION_CREDENTIALS"]).read()),
        ["<https://www.googleapis.com/auth/cloud-platform>"]
    )
    
    # Add IAP audience
    credentials = credentials.with_audience(
        f"<https://iap.googleapis.com/projects/{os.environ[>"GOOGLE_CLOUD_PROJECT"]}/global/backendServices/YOUR_BACKEND_SERVICE_ID"
    )
    
    request = Request()
    credentials.refresh(request)
    
    print("Token generation successful")
    print(f"Token type: {type(credentials.token)}")
    print(f"Token length: {len(credentials.token) if credentials.token else 0}")
    
    # Set headers with the new token
    headers = {
        "Authorization": f"Bearer {credentials.token}"
    }
    
    with open(os.environ["GITHUB_ENV"], "a") as f:
        f.write(f"PREFECT_CLIENT_CUSTOM_HEADERS={json.dumps(headers)}\n")
    '
Could you please: 1. Replace 
YOUR_BACKEND_SERVICE_ID
with your actual backend service ID from Cloud Run 2. Run both scripts 3. Share the output (excluding any sensitive information) This will help us: 1. Verify the correct URL is being used 2. Ensure the IAP token is being generated with the correct audience 3. See any additional error information from the server Also, could you verify: 1. Can you access the Prefect UI through your browser? 2. What's the exact value of your 
PREFECT_API_URL
environment variable?
u
@Marvin I've run the test again with the url fixed Run python -c ' python -c ' import requests import os import json headers = json.loads(os.environ["PREFECT_CLIENT_CUSTOM_HEADERS"]) response = requests.get( "https://prefect-server-1030217702417.asia-northeast3.run.app/api/health", headers=headers ) print(f"Status code: {response.status_code}") print(f"Response: {response.text if response.ok else "Failed"}") ' shell: /usr/bin/bash -e {0} env: PREFECT_API_URL: https://prefect-server-1030217702417.asia-northeast3.run.app/api PREFECT_CLIENT_CSRF_SUPPORT_ENABLED: false pythonLocation: /opt/hostedtoolcache/Python/3.13.5/x64 PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.13.5/x64/lib/pkgconfig Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.13.5/x64 Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.13.5/x64 Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.13.5/x64 LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.13.5/x64/lib CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /home/runner/work/ml-data-pipeline/ml-data-pipeline/gha-creds-f61a23eeb72710aa.json GOOGLE_APPLICATION_CREDENTIALS: /home/runner/work/ml-data-pipeline/ml-data-pipeline/gha-creds-f61a23eeb72710aa.json GOOGLE_GHA_CREDS_PATH: /home/runner/work/ml-data-pipeline/ml-data-pipeline/gha-creds-f61a23eeb72710aa.json CLOUDSDK_CORE_PROJECT: solvook-infra CLOUDSDK_PROJECT: solvook-infra GCLOUD_PROJECT: solvook-infra GCP_PROJECT: solvook-infra GOOGLE_CLOUD_PROJECT: solvook-infra PREFECT_CLIENT_CUSTOM_HEADERS: {"Authorization":"***"} Status code: 401 Response: Failed
m
✅ thought for 13.9 seconds
I notice that we need to specifically handle the IAP audience for the token. For Cloud Run with IAP, the token needs a specific audience format. Let's modify the token generation:
Copy code
yaml
- name: Generate IAP Token
  run: |
    python -c '
    from google.oauth2 import service_account
    from google.auth.transport.requests import Request
    import os
    import json
    
    # Load credentials
    credentials = service_account.Credentials.from_service_account_info(
        json.loads(open(os.environ["GOOGLE_APPLICATION_CREDENTIALS"]).read()),
        ["<https://www.googleapis.com/auth/cloud-platform>"]
    )
    
    # For Cloud Run with IAP, the audience format is:
    # <https://YOUR_REGION-run.googleapis.com/projects/PROJECT_NUMBER/locations/REGION/services/SERVICE_NAME>
    
    project_number = "1030217702417"  # From your URL
    region = "asia-northeast3"        # From your URL
    service_name = "prefect-server"   # From your URL
    
    audience = f"https://{region}-<http://run.googleapis.com/projects/{project_number}/locations/{region}/services/{service_name}|run.googleapis.com/projects/{project_number}/locations/{region}/services/{service_name}>"
    
    # Add the correct audience
    credentials = credentials.with_audience(audience)
    
    request = Request()
    credentials.refresh(request)
    
    print("Token details:")
    print(f"Audience: {audience}")
    print(f"Token length: {len(credentials.token) if credentials.token else 0}")
    
    # Set headers with the new token
    headers = {
        "Authorization": f"Bearer {credentials.token}"
    }
    
    with open(os.environ["GITHUB_ENV"], "a") as f:
        f.write(f"PREFECT_CLIENT_CUSTOM_HEADERS={json.dumps(headers)}\n")
    '

- name: Test IAP Authentication
  run: |
    python -c '
    import requests
    import os
    import json
    
    url = "<https://prefect-server-1030217702417.asia-northeast3.run.app/api/health>"
    headers = json.loads(os.environ["PREFECT_CLIENT_CUSTOM_HEADERS"])
    
    print(f"Testing URL: {url}")
    print(f"Headers present: {list(headers.keys())}")
    
    response = requests.get(
        url,
        headers=headers,
        timeout=30
    )
    
    print(f"\nStatus code: {response.status_code}")
    if not response.ok:
        print(f"Response headers: {dict(response.headers)}")
        print(f"Response body: {response.text}")
    '
The key changes are: 1. Added proper audience format for Cloud Run with IAP 2. Using the correct project number, region, and service name from your URL 3. Added more detailed error output Could you try this updated version? Also, to verify: 1. Can you confirm if these values are correct: - Project number: 1030217702417 - Region: asia-northeast3 - Service name: prefect-server 2. Can you access the Prefect UI through your browser with IAP authentication? If you're able to access the UI but the API calls are failing, we might need to check: 1. The IAM roles assigned to the service account 2. The OAuth consent screen configuration 3. The IAP settings in Cloud Console
u
@Marvin I'm already generating iap token from github action
Copy code
- id: 'auth'
        uses: 'google-github-actions/auth@v2'
        with:
          project_id: '***'
          workload_identity_provider: 'projects/***/locations/global/workloadIdentityPools/ml-team-pool/providers/github-action'
          service_account: 'ml-github-sa@***.<http://iam.gserviceaccount.com|iam.gserviceaccount.com>'
          token_format: 'id_token'
          id_token_audience: '<https://prefect-server>-***.<http://asia-northeast3.run.app|asia-northeast3.run.app>'
m
✅ thought for 12.7 seconds
Ah, I see! You're using Workload Identity Federation with GitHub Actions and generating an ID token. This is a different approach from what I suggested earlier. Let's modify the setup to use this ID token correctly. When using 
google-github-actions/auth
with 
token_format: 'id_token'
, we need to set the token differently in the Prefect headers. Here's how we should modify your workflow:
Copy code
yaml
- id: 'auth'
  uses: 'google-github-actions/auth@v2'
  with:
    project_id: '***'
    workload_identity_provider: 'projects/***/locations/global/workloadIdentityPools/ml-team-pool/providers/github-action'
    service_account: 'ml-github-sa@***.<http://iam.gserviceaccount.com|iam.gserviceaccount.com>'
    token_format: 'id_token'
    id_token_audience: '<https://prefect-server>-***.<http://asia-northeast3.run.app|asia-northeast3.run.app>'

- name: Set Prefect Headers
  run: |
    python -c '
    import os
    import json
    
    # Get the ID token from the previous step
    id_token = """${{ steps.auth.outputs.id_token }}"""
    
    # Set the headers with the ID token
    headers = {
        "Authorization": f"Bearer {id_token}"
    }
    
    # Set for Prefect client
    with open(os.environ["GITHUB_ENV"], "a") as f:
        f.write(f"PREFECT_CLIENT_CUSTOM_HEADERS={json.dumps(headers)}\n")
    '

- name: Test IAP Authentication
  run: |
    python -c '
    import requests
    import os
    import json
    
    url = "${{ secrets.PREFECT_API_URL }}/health"
    headers = json.loads(os.environ["PREFECT_CLIENT_CUSTOM_HEADERS"])
    
    print(f"Testing URL: {url}")
    print(f"Headers present: {list(headers.keys())}")
    
    try:
        response = requests.get(
            url,
            headers=headers,
            timeout=30
        )
        
        print(f"\nStatus code: {response.status_code}")
        if not response.ok:
            print(f"Response headers: {dict(response.headers)}")
    except Exception as e:
        print(f"Request failed: {str(e)}")
    '

- name: Setup Prefect
  run: |
    prefect config set PREFECT_API_URL=${{ secrets.PREFECT_API_URL }}
Key changes: 1. Using the ID token directly from the 
google-github-actions/auth
output 2. Setting it as a Bearer token in the 
Authorization
header 3. No need for additional token generation since we're using the ID token from the auth action Also, please verify: 1. The IAP settings in Cloud Console match the 
id_token_audience
you've configured 2. The service account has the necessary IAP-secured Web App User role (
roles/iap.httpsResourceAccessor
) 3. The OAuth consent screen is properly configured for the domain Would you like me to provide more details about any of these aspects?
u
@Marvin and also, i'm already setting 
PREFECT_CLIENT_CUSTOM_HEADERS
from github action like below
Copy code
- name: Set Prefect Custom Headers with ID Token
        run: |
          unset PREFECT_API_KEY
          echo "PREFECT_CLIENT_CUSTOM_HEADERS={\"Authorization\":\"Bearer ${{ steps.auth.outputs.id_token }}\"}" >> $GITHUB_ENV
m
✅ thought for 18.3 seconds
I see! You're already setting up the 
PREFECT_CLIENT_CUSTOM_HEADERS
correctly. Let's debug why it's still returning a 401 error. First, let's verify the headers are being set and passed correctly:
Copy code
yaml
- name: Debug Headers and Test Connection
  run: |
    python -c '
    import os
    import json
    import requests
    from urllib.parse import urljoin
    
    # Debug current environment
    headers = json.loads(os.environ.get("PREFECT_CLIENT_CUSTOM_HEADERS", "{}"))
    api_url = os.environ.get("PREFECT_API_URL", "")
    
    print("Debug Information:")
    print(f"API URL: {api_url}")
    print(f"Headers present: {list(headers.keys())}")
    print(f"Authorization header starts with: {headers.get('Authorization', '')[:20]}...")
    
    # Test basic endpoint
    health_url = urljoin(api_url, "health")
    print(f"\nTesting health endpoint: {health_url}")
    
    try:
        response = requests.get(
            health_url,
            headers=headers,
            timeout=30
        )
        
        print(f"Status code: {response.status_code}")
        print(f"Response headers: {dict(response.headers)}")
        
        if response.status_code == 401:
            print("\nPossible issues:")
            print("1. Token might be expired")
            print("2. Token audience might not match")
            print("3. IAP might be expecting different header format")
            
    except Exception as e:
        print(f"Request failed: {str(e)}")
    '
Also, let's verify the complete workflow order:
Copy code
yaml
- uses: 'google-github-actions/auth@v2'
  id: 'auth'
  with:
    project_id: '***'
    workload_identity_provider: 'projects/***/locations/global/workloadIdentityPools/ml-team-pool/providers/github-action'
    service_account: 'ml-github-sa@***.<http://iam.gserviceaccount.com|iam.gserviceaccount.com>'
    token_format: 'id_token'
    id_token_audience: '<https://prefect-server>-***.<http://asia-northeast3.run.app|asia-northeast3.run.app>'

- name: Set Prefect Custom Headers with ID Token
  run: |
    unset PREFECT_API_KEY
    echo "PREFECT_CLIENT_CUSTOM_HEADERS={\"Authorization\":\"Bearer ${{ steps.auth.outputs.id_token }}\"}" >> $GITHUB_ENV

- name: Configure Prefect
  run: |
    prefect config set PREFECT_API_URL=${{ secrets.PREFECT_API_URL }}
    
    # Debug Prefect configuration
    echo "Current Prefect Configuration:"
    prefect config view
Could you also try making a direct curl request to verify the token:
Copy code
yaml
- name: Test with curl
  run: |
    # Test the health endpoint
    curl -v -H "Authorization: Bearer ${{ steps.auth.outputs.id_token }}" ${{ secrets.PREFECT_API_URL }}/health
Questions to help debug: 1. Can you confirm that the 
id_token_audience
exactly matches your Cloud Run service URL? 2. Are you able to access other IAP-protected endpoints using the same token? 3. Have you tried accessing the Prefect API endpoint directly through curl or Postman with the same token to verify it works outside of Prefect? 4. Could you check the Cloud Run logs to see if there are any specific error messages when the 401 occurs? This will help us understand: - If the token is being properly generated and formatted - If the token is valid for your IAP configuration - If there's an issue specific to how Prefect is handling the headers - If there might be an issue with the IAP configuration itself
u
@Marvin Here's curl response
Copy code
> GET /api/health HTTP/2
> Host: prefect-server-***.<http://asia-northeast3.run.app|asia-northeast3.run.app>
> User-Agent: curl/8.5.0
> Accept: */*
> Authorization: ***
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [282 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [282 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 401 
< x-goog-iap-generated-response: true
< date: Sun, 20 Jul 2025 14:40:48 GMT
< content-type: text/html
< server: Google Frontend
< content-length: 57
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
< 
{ [57 bytes data]
100    57  100    57    0     0    243      0 --:--:-- --:--:-- --:--:--   243
* Connection #0 to host prefect-server-***.<http://asia-northeast3.run.app|asia-northeast3.run.app> left intact
Invalid IAP credentials: JWT 'email' claim isn't a string
m
✅ thought for 10.1 seconds
Ah! The error message is very helpful: 
Invalid IAP credentials: JWT 'email' claim isn't a string
. This indicates that the ID token being generated doesn't have the email claim that IAP expects. When using IAP with ID tokens, we need to include specific claims in the token. Let's modify your auth configuration:
Copy code
yaml
- uses: 'google-github-actions/auth@v2'
  id: 'auth'
  with:
    project_id: '***'
    workload_identity_provider: 'projects/***/locations/global/workloadIdentityPools/ml-team-pool/providers/github-action'
    service_account: 'ml-github-sa@***.<http://iam.gserviceaccount.com|iam.gserviceaccount.com>'
    token_format: 'id_token'
    id_token_audience: '<https://prefect-server>-***.<http://asia-northeast3.run.app|asia-northeast3.run.app>'
    # Add these claims that IAP requires
    id_token_include_email: true
    additional_claims: |
      {
        "email": "${{ vars.SERVICE_ACCOUNT_EMAIL }}",
        "sub": "${{ vars.SERVICE_ACCOUNT_EMAIL }}"
      }

- name: Set Prefect Custom Headers with ID Token
  run: |
    unset PREFECT_API_KEY
    echo "PREFECT_CLIENT_CUSTOM_HEADERS={\"Authorization\":\"Bearer ${{ steps.auth.outputs.id_token }}\"}" >> $GITHUB_ENV
Key changes: 1. Added 
id_token_include_email: true
to include the email claim 2. Added 
additional_claims
to explicitly set the email and subject claims 3. Using the service account email as both the email and subject claims Make sure to: 1. Set the 
SERVICE_ACCOUNT_EMAIL
variable in your GitHub repository variables with the full service account email (
ml-github-sa@***.<http://iam.gserviceaccount.com|iam.gserviceaccount.com>
) 2. Verify that the service account has the 
roles/iap.httpsResourceAccessor
role This should resolve the "JWT 'email' claim isn't a string" error from IAP. Would you like to try this modified configuration?
u
@Marvin I've done for only Key Changes 1. and heres result below
Copy code
> GET /api/health HTTP/2
> Host: prefect-server-***.<http://asia-northeast3.run.app|asia-northeast3.run.app>
> User-Agent: curl/8.5.0
> Accept: */*
> Authorization: ***
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [282 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [282 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< HTTP/2 401 
< x-goog-iap-generated-response: true
< date: Sun, 20 Jul 2025 14:43:57 GMT
< content-type: text/html
< server: Google Frontend
< content-length: 68
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
< 
{ [68 bytes data]
100    68  100    68    0     0    177      0 --:--:-- --:--:-- --:--:--   177
* Connection #0 to host prefect-server-***.<http://asia-northeast3.run.app|asia-northeast3.run.app> left intact
Invalid IAP credentials: Invalid bearer token. Invalid JWT audience.
m
✅ thought for 10.3 seconds
Ah, now we're getting a different error: 
Invalid IAP credentials: Invalid bearer token. Invalid JWT audience
. This means we need to adjust the 
id_token_audience
format. For Google Cloud IAP, the audience should follow a specific format. Let's modify your auth configuration:
Copy code
yaml
- uses: 'google-github-actions/auth@v2'
  id: 'auth'
  with:
    project_id: '***'
    workload_identity_provider: 'projects/***/locations/global/workloadIdentityPools/ml-team-pool/providers/github-action'
    service_account: 'ml-github-sa@***.<http://iam.gserviceaccount.com|iam.gserviceaccount.com>'
    token_format: 'id_token'
    id_token_include_email: true
    # Update the audience format for IAP
    id_token_audience: '/projects/YOUR_PROJECT_NUMBER/global/backendServices/YOUR_BACKEND_SERVICE_ID'

- name: Set Prefect Custom Headers with ID Token
  run: |
    unset PREFECT_API_KEY
    echo "PREFECT_CLIENT_CUSTOM_HEADERS={\"Authorization\":\"Bearer ${{ steps.auth.outputs.id_token }}\"}" >> $GITHUB_ENV
To get the correct audience value: 1. Go to the Google Cloud Console 2. Navigate to Security → Identity-Aware Proxy 3. Find your Cloud Run service 4. Click on the "OAuth 2.0 Client ID" link 5. In the URL or details, you should see the Backend Service ID Alternatively, you can get it via gcloud:
Copy code
bash
gcloud iap oauth-clients list --project=YOUR_PROJECT_ID
The audience format should be:
Copy code
/projects/PROJECT_NUMBER/global/backendServices/BACKEND_SERVICE_ID
Could you: 1. Get your Backend Service ID from the IAP settings 2. Update the 
id_token_audience
with the correct format 3. Try the request again This should resolve the "Invalid JWT audience" error. Would you like me to help you find the correct Backend Service ID?
2 Views
Open in Slack
PreviousNext
Powered by