https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I want your help to fix and properly ...
# marvin-ai
m
@Marvin I want your help to fix and properly enhance my stack, here's my compose file:
Copy code
name: prefect-wizard

services:
  nginx:
    image: nginx:latest
    ports:
      - "80:80" # Expose NGINX on port 80
    volumes:
      - ./proxies/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - server
      - api
      - keycloak
      - oauth2-proxy

  # VNC browser for local development to use for keycloak and oauth2-proxy
  browser:
    image: jlesage/firefox
    ports:
      - "5800:5800"
    environment:
      # Optional: Set a password for the VNC web interface
      - VNC_PASSWORD=my_password
    depends_on:
      - keycloak
      - oauth2-proxy
      - nginx

  keycloak:
    image: <http://quay.io/keycloak/keycloak:24.0|quay.io/keycloak/keycloak:24.0>
    command: [ "start-dev", "--http-port=8080" ]
    environment:
      KC_BOOTSTRAP_ADMIN_USERNAME: admin
      KC_BOOTSTRAP_ADMIN_PASSWORD: admin
      #KC_HOSTNAME: "keycloak"          # no path, just host (and optional port)
      KC_HOSTNAME_STRICT: "false" # dev convenience
      KC_HOSTNAME_STRICT_HTTPS: "false" # dev convenience
      KC_PROXY: "edge" # recommended when behind a proxy
    ports:
      - "8080:8080"
    volumes:
      - keycloak_data:/opt/keycloak/data
      - ./keycloak/local/config.json:/opt/keycloak/conf/realm-config.json

  oauth2-proxy:
    image: <http://quay.io/oauth2-proxy/oauth2-proxy:v7.5.1|quay.io/oauth2-proxy/oauth2-proxy:v7.5.1>
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "4180:4180"
    command: [ "--config=/etc/oauth2-proxy.cfg" ]
    volumes:
      - "./proxies/oauth2-proxy.cfg:/etc/oauth2-proxy.cfg"
    depends_on:
      - keycloak


  api:
    build:
      context: .
      dockerfile: docker/api/nonprod/Dockerfile
    ports:
      - "9832:9832"
    environment:
      PREFECT_API_URL: <http://server:4200/api>
    volumes:
      - ./api:/app
      - ./autoupdate:/app/autoupdate
      - ./core:/app/core
    extra_hosts:
      - "host.docker.internal:host-gateway"
    env_file:
      - .env
    restart: always
    healthcheck:
      test: [ "CMD-SHELL", "curl -f <http://localhost:9832/health> || exit 1" ]
      interval: 5s
      timeout: 5s
      retries: 5

  db:
    ...

  redis:
    ...

  server:
    image: prefecthq/prefect:3.4.13-python3.11
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@db:5432/prefect>
      PREFECT_SERVER_API_HOST: 0.0.0.0
      PREFECT_UI_API_URL: "<http://localhost:4200/api>"
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
      PREFECT_REDIS_MESSAGING_DB: 0
      PREFECT_LOGGING_LEVEL: DEBUG
      PREFECT_LOGGING_LOG_PRINTS: True
      PREFECT_SERVER_LOGGING_LEVEL: DEBUG
      PREFECT_SERVER_LOGGING_LOG_PRINTS: True
      #PREFECT_SERVER_API_AUTH_STRING: "admin:admin"
    command: prefect server start --no-services
    ports:
      - "4200:4200"
    healthcheck:
      test: [ "CMD-SHELL", "python -c 'import json,urllib.request,sys; sys.exit(0 if json.loads(urllib.request.urlopen(\"<http://localhost:4200/api/health>\", timeout=1).read()) is True else 1)' " ]
      interval: 5s
      timeout: 3s
      retries: 30

  services:
    ...

  create-deployment:
    ...

  autoupdate-worker:
    image: prefect-workpool
    depends_on:
      server:
        condition: service_healthy
      services:
        condition: service_started
      create-deployment:
        condition: service_completed_successfully
    environment:
      PREFECT_API_URL: <http://server:4200/api>
      DOCKER_HOST: <tcp://host.docker.internal:2375>
      PREFECT_LOCAL_STORAGE_PATH: /logs/storage
      # DOCKER_TLS_VERIFY: ${DOCKER_TLS_VERIFY:-}
      # DOCKER_CERT_PATH: ${DOCKER_CERT_PATH:-}
    restart: always
    volumes:
      - ./autoupdate:/opt/prefect/autoupdate
      - ./core:/opt/prefect/autoupdate/core
      - ${HOME}/.wizard:/logs
    extra_hosts:
      - "host.docker.internal:host-gateway"
    env_file:
      - .env

volumes:
  postgres_data:
  redis_data:
  keycloak_data:
I will send you keycloak config, oauth-proxy configs, nginx configs on separate messages as i am not allowed to post a message that is too long The error i am gettting: When I try to visit http://nginx on my virtual browser, it routes me to http://keycloak:8080/realms/wizard/protocol/.... with error 
invalid parameter: redirect uri
m
✅ thought for 97.8 seconds
Thanks for the details — the Keycloak error you’re seeing almost always means the 
redirect_uri
sent in the OIDC authorization request does not exactly match any value in the Keycloak client’s “Valid Redirect URIs”. In your setup, you’re accessing via the Docker-internal hostname 
<http://nginx>
(from the 
browser
container). That means: - oauth2-proxy must advertise its redirect URL as 
<http://nginx/oauth2/callback>
(since that’s what the browser sees) - The Keycloak client for oauth2-proxy must allow that exact redirect URL - NGINX must proxy 
/oauth2/*
to the oauth2-proxy service and pass X-Forwarded headers so oauth2-proxy can compute the correct external URL Quick fix checklist: 1) In Keycloak (realm: wizard) for the oauth2-proxy client: - Valid Redirect URIs: - Add 
<http://nginx/oauth2/callback>
- If you also plan to hit via your host browser at http://localhost, add 
<http://localhost/oauth2/callback>
too - Web Origins: - Add 
<http://nginx>
(or 
+
for dev) - Client type typically “confidential” with a client secret that matches oauth2-proxy 2) In oauth2-proxy.cfg include: - 
redirect_url = "<http://nginx/oauth2/callback>"
- 
oidc_issuer_url = "<http://keycloak:8080/realms/wizard>"
- 
client_id
and 
client_secret
matching the Keycloak client - 
reverse_proxy = true
- For HTTP dev: 
cookie_secure = false
- Optional but recommended for NGINX auth_request: 
set_xauthrequest = true
, 
pass_authorization_header = true
Example oauth2-proxy.cfg
Copy code
provider = "oidc"
oidc_issuer_url = "<http://keycloak:8080/realms/wizard>"

client_id = "oauth2-proxy"
client_secret = "<your-client-secret>"

# Must match what the browser sees externally
redirect_url = "<http://nginx/oauth2/callback>"

email_domains = [ "*" ]
cookie_secret = "<32-byte-base64-secret>"
cookie_secure = false
cookie_samesite = "lax"
reverse_proxy = true
set_xauthrequest = true
pass_authorization_header = true
3) In NGINX, be sure you: - Proxy 
/oauth2/
to 
oauth2-proxy:4180
- Use 
auth_request
to protect your upstream(s) - Forward the X-Forwarded headers so oauth2-proxy knows the external host (nginx) and scheme Example nginx.conf snippet
Copy code
events {}
http {
  upstream prefect_ui {
    server server:4200;
  }
  upstream oauth2_proxy {
    server oauth2-proxy:4180;
  }

  server {
    listen 80;
    server_name nginx;

    # oauth2-proxy endpoints
    location /oauth2/ {
      proxy_pass <http://oauth2_proxy>;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Scheme $scheme;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Uri $request_uri;
    }

    # auth gate used by auth_request
    location = /oauth2/auth {
      proxy_pass <http://oauth2_proxy>;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Scheme $scheme;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Uri $request_uri;
    }

    # Protect Prefect UI (example)
    location / {
      auth_request /oauth2/auth;
      error_page 401 = /oauth2/sign_in;

      # optional: pass user info to upstream
      auth_request_set $user $upstream_http_x_auth_request_user;
      proxy_set_header X-User $user;

      proxy_pass <http://prefect_ui>;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
  }
}
Why this matters:
- oauth2-proxy derives the 
redirect_uri
from its 
--redirect-url
(or the forwarded host/scheme). If that ends up as 
<http://oauth2-proxy:4180/oauth2/callback>
or 
<http://localhost/oauth2/callback>
but your Keycloak client only allows 
<http://nginx/oauth2/callback>
, Keycloak will throw “invalid parameter: redirect uri”. A couple of clarifying questions so I can tailor this exactly: - Which hostname(s) do you plan to access from your actual browser: 
<http://nginx>
, 
<http://localhost>
, or a real domain? - What’s your current oauth2-proxy.cfg? - For the Keycloak client used by oauth2-proxy, what are the current “Valid Redirect URIs” and “Web Origins”? - Could you share your nginx.conf server block(s) for the Prefect routes and /oauth2/*? Relevant docs: - Keycloak redirect URIs: Keycloak docs: Redirect URIs - oauth2-proxy configuration: oauth2-proxy config overview - oauth2-proxy with NGINX auth_request: NGINX auth_request setup Send over your Keycloak, oauth2-proxy, and NGINX configs when you can, and I’ll validate them line-by-line.
m
@Marvin I fixed redirect issue and now it works The issue i am getting now, after successfully logging in with keycloak, i am redirected to oauth2-proxy 403 page with error: unable to find CSRF TOKEN oauth2-proxy config:
Copy code
provider = "keycloak-oidc"
oidc_issuer_url = "<http://keycloak:8080/realms/wizard>"
client_id = "wizard-oauth"
client_secret = "olcWi1lGcyrU82MtdvNgK8Nm7x4C8ieA"
cookie_secret = "bHEyY2E0dEZHSTlnY2hJa0FRWnRERURxaDg4Qm5qY3k="
email_domains = "*"
insecure_oidc_allow_unverified_email = true
http_address = "0.0.0.0:4180"
redirect_url = "<http://nginx:80/oauth2/callback>"
pass_access_token = true
skip_provider_button = true
reverse_proxy = true
nginx:
Copy code
events {
  worker_connections 1024;
}

http {
  upstream prefect_ui {
    server server:4200;
  }
  upstream oauth2_proxy {
    server oauth2-proxy:4180;
  }
  upstream api {
    server api:9832;
  }

  server {
    listen 80;
    server_name nginx;

    # The /oauth2/ location handles all OAuth2 Proxy traffic
    location /oauth2/ {
      proxy_pass <http://oauth2_proxy>;
      proxy_set_header Host                    $host;
      proxy_set_header X-Real-IP               $remote_addr;
      proxy_set_header X-Scheme                $scheme;
      proxy_set_header X-Auth-Request-Redirect $request_uri;
    }

    # The /oauth2/auth location is the actual authentication endpoint
    location = /oauth2/auth {
      proxy_pass       <http://oauth2_proxy>;
      proxy_set_header Host             $host;
      proxy_set_header X-Real-IP        $remote_addr;
      proxy_set_header X-Scheme         $scheme;
      proxy_set_header Content-Length   "";
      proxy_pass_request_body           off;
    }

    # The / location is for the main Prefect UI
    location / {
      auth_request /oauth2/auth;
      error_page 401 = /oauth2/sign_in;

      auth_request_set $user  $upstream_http_x_auth_request_user;
      auth_request_set $email $upstream_http_x_auth_request_email;
      proxy_set_header X-User $user;
      proxy_set_header X-Email $email;

      # if you enabled --pass-access-token, this will pass the token to the backend
      auth_request_set $token  $upstream_http_x_auth_request_access_token;
      proxy_set_header X-Access-Token $token;

      proxy_pass <http://prefect_ui>;
    }
...
m
✅ thought for 80.3 seconds
Great progress getting the redirect fixed. The “unable to find CSRF token” from oauth2-proxy almost always means the CSRF cookie was not set or not sent back on the callback. In your setup, this is typically due to using HTTP with a Secure cookie or missing forwarded headers. Do these two changes: 1) Update oauth2-proxy.cfg - Add cookie settings and xauthrequest headers - Optional: drop the explicit :80 on the redirect URL (not required, but keeps it consistent)
Copy code
provider = "keycloak-oidc"
oidc_issuer_url = "<http://keycloak:8080/realms/wizard>"

client_id = "wizard-oauth"
client_secret = "olcWi1lGcyrU82MtdvNgK8Nm7x4C8ieA"

redirect_url = "<http://nginx/oauth2/callback>"  # (port 80 implied)
reverse_proxy = true

email_domains = "*"
insecure_oidc_allow_unverified_email = true
pass_access_token = true

cookie_secret = "bHEyY2E0dEZHSTlnY2hJa0FRWnRERURxaDg4Qm5qY3k="
cookie_secure = false           # IMPORTANT for HTTP; otherwise the cookie won’t be sent
cookie_samesite = "lax"

set_xauthrequest = true
pass_authorization_header = true
2) Strengthen NGINX forwarded headers - Ensure oauth2-proxy sees the external host/scheme so it sets cookies for the right host and computes redirects correctly - Also set X-Auth-Request-Redirect on the /oauth2/auth location
Copy code
server {
  listen 80;
  server_name nginx;

  location /oauth2/ {
    proxy_pass <http://oauth2_proxy>;
    proxy_set_header Host                  $host;
    proxy_set_header X-Real-IP             $remote_addr;
    proxy_set_header X-Forwarded-For       $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto     $scheme;
    proxy_set_header X-Forwarded-Host      $host;
    proxy_set_header X-Forwarded-Uri       $request_uri;
  }

  location = /oauth2/auth {
    proxy_pass                     <http://oauth2_proxy>;
    proxy_set_header Host          $host;
    proxy_set_header X-Real-IP     $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host  $host;
    proxy_set_header X-Forwarded-Uri   $request_uri;

    proxy_set_header Content-Length "";
    proxy_pass_request_body off;

    proxy_set_header X-Auth-Request-Redirect $request_uri;
  }

  location / {
    auth_request /oauth2/auth;
    error_page 401 = /oauth2/sign_in;

    auth_request_set $user  $upstream_http_x_auth_request_user;
    auth_request_set $email $upstream_http_x_auth_request_email;
    proxy_set_header X-User  $user;
    proxy_set_header X-Email $email;

    auth_request_set $token  $upstream_http_x_auth_request_access_token;
    proxy_set_header X-Access-Token $token;

    proxy_pass <http://prefect_ui>;
    proxy_set_header Host                 $host;
    proxy_set_header X-Forwarded-For      $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto    $scheme;
    proxy_set_header X-Forwarded-Host     $host;
  }
}
Then: - Clear cookies for the nginx host in your VNC browser - Hit http://nginx/oauth2/sign_in, login, and verify you no longer hit the 403 If it still fails, please: - Check in the browser devtools if a CSRF cookie from oauth2-proxy is ever set for host nginx before the redirect (name often like _oauth2_proxy_csrf or __Host-oauth2-proxy-csrf) - Share the oauth2-proxy logs around the login; we can turn up logging if needed References: - oauth2-proxy CSRF troubleshooting: Unable to find a valid CSRF token - oauth2-proxy config overview: Configuration overview Once auth is working, we can harden it (cookie_expire, refresh, AllowedGroups from Keycloak, etc.) and make sure Prefect UI/API routes and websockets behave nicely behind NGINX.
m
@Marvin Getting a different error now: Here's oauth2-proxy container logs:: nginx GET - "/oauth2/sign_in?rd=%2Foauth2%2Fcallback%3Fstate%3DlUEER1IILI1G4fp4WSLssD4TzADSIAWCh3TtgSmkfOk%253A%252Foauth2%252Fcallback%3F%26session_state%3Dcd295beb-7552-4ea7-9df3-17bd91045850%26iss%3Dhttp%253A%252F%252Fkeycloak%253A8080%252Frealms%252Fwizard%26code%3D1b2a0413-48ac-49b8-a316-a852b7a87e2d.cd295beb-7552-4ea7-9df3-17bd91045850.e954fcbe-a31c-41ee-8c6b-da7f80aafa50" HTTP/1.0 "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0" 302 709 0.000 [2025/08/30 225939] [oauthproxy.go:841] Error creating session during OAuth2 callback: audience from claim aud with value [account] does not match with any of allowed audiences map[wizard-oauth:{}]
m
✅ thought for 43.8 seconds
Nice—now we’re just hitting an audience validation mismatch. What the log means - oauth2-proxy expects the token’s audience to include your client_id (wizard-oauth). This is shown by allowed audiences map[wizard-oauth:{}]. - Keycloak is returning tokens with aud = [ "account" ] instead, so oauth2-proxy rejects the callback. Two clean ways to fix it (pick one) Option A: Tell oauth2-proxy to use Keycloak’s azp claim - Keycloak sets azp (Authorized Party) to the actual client_id the token was issued for (wizard-oauth). - Configure oauth2-proxy to validate against azp instead of aud. Add this to oauth2-proxy.cfg and restart the container:
Copy code
# existing settings...
provider = "keycloak-oidc"
oidc_issuer_url = "<http://keycloak:8080/realms/wizard>"
client_id = "wizard-oauth"
client_secret = "REDACTED"
redirect_url = "<http://nginx/oauth2/callback>"
reverse_proxy = true

email_domains = [ "*" ]
insecure_oidc_allow_unverified_email = true
pass_access_token = true

cookie_secret = "REDACTED"
cookie_secure = false
cookie_samesite = "lax"
set_xauthrequest = true
pass_authorization_header = true

# KEY FIX:
oidc_audience_claim = "azp"
Option B: Make Keycloak include your client as an audience - In Keycloak Admin: - Clients > wizard-oauth > Client scopes (or Mappers) - Create mapper: “Audience” - Included Client Audience: wizard-oauth - Add to access token: ON - Add to ID token: ON - This ensures aud includes wizard-oauth, satisfying oauth2-proxy’s default behavior. After applying either fix - Clear cookies for the nginx host in your VNC Firefox - Try again via http://nginx/ Quick checks - Decode the ID token you receive from Keycloak and confirm: - azp == wizard-oauth - aud contains wizard-oauth (if you applied Option B) - Ensure oauth2-proxy still sees the correct external host/scheme via the NGINX forwarded headers you set earlier. Security note: you posted a client_secret in the thread—rotate it in Keycloak and update your oauth2-proxy config. Useful docs - oauth2-proxy OIDC options (incl. oidc_audience_claim): oauth2-proxy configuration overview - Keycloak audience mapping (concept): Keycloak docs: Audience If it still errors, share: - A redacted decoded ID token’s header/payload (remove sig), so we can verify aud/azp - The exact oauth2-proxy log line after your change
m
@Marvin I used option 2 as there's no such config called: "oidc_audience_claim" Now, after logging in, I see these logs on the oauth2 proxy:
Copy code
172.20.0.10 - 883f9412-b5fd-4dc1-956c-c3f1117ae167 - test@test.com [2025/08/30 23:14:15] [AuthSuccess] Authenticated via OAuth2: Session{email:test@test.com user:f6246d8d-cade-450b-ba3d-74ca051da430 PreferredUsername:test token:true id_token:true created:2025-08-30 23:14:15.378336396 +0000 UTC m=+42.703549616 expires:2025-08-30 23:19:15.366800858 +0000 UTC m=+342.692014082 refresh_token:true groups:[role:default-roles-wizard role:offline_access role:uma_authorization role:account:manage-account role:account:manage-account-links role:account:view-profile]}

172.20.0.10 - 883f9412-b5fd-4dc1-956c-c3f1117ae167 - - [2025/08/30 23:14:15] nginx GET - "/oauth2/callback?state=K3JplFChZU1FUmKwVJvtvPrnwNi0ffkPsenrFm8_Urc%3A%2F&session_state=1e14678d-c3f7-4311-87cf-955bad92ce45&iss=http%3A%2F%2Fkeycloak%3A8080%2Frealms%2Fwizard&code=f94e23eb-83ae-4346-945b-f3c22f965b63.1e14678d-c3f7-4311-87cf-955bad92ce45.e954fcbe-a31c-41ee-8c6b-da7f80aafa50" HTTP/1.0 "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0" 302 24 0.026
Nginx however gives 502 bad gateway! Here's nginx configs:
Copy code
# See reference:
# <https://github.com/pasha-r/oauth2_proxy/blob/master/docs/configuration/configuration.md#config-file>

events {
  worker_connections 1024;
}

http {
  upstream prefect_ui {
    server server:4200;
  }
  upstream oauth2_proxy {
    server oauth2-proxy:4180;
  }
  upstream api {
    server api:9832;
  }

  server {
    listen 80;
    server_name nginx;

    # The /oauth2/ location handles all OAuth2 Proxy traffic
    location /oauth2/ {
      proxy_pass <http://oauth2_proxy>;
      proxy_set_header Host                    $host;
      proxy_set_header X-Real-IP               $remote_addr;
      proxy_set_header X-Scheme                $scheme;
      proxy_set_header X-Auth-Request-Redirect $request_uri;
    }

    # The /oauth2/auth location is the actual authentication endpoint
    location = /oauth2/auth {
      proxy_pass       <http://oauth2_proxy>;
      proxy_set_header Host             $host;
      proxy_set_header X-Real-IP        $remote_addr;
      proxy_set_header X-Scheme         $scheme;
      proxy_set_header Content-Length   "";
      proxy_pass_request_body           off;
    }

    # The / location is for the main Prefect UI
    location / {
      auth_request /oauth2/auth;
      error_page 401 = /oauth2/sign_in;

      auth_request_set $user  $upstream_http_x_auth_request_user;
      auth_request_set $email $upstream_http_x_auth_request_email;
      proxy_set_header X-User $user;
      proxy_set_header X-Email $email;

      # if you enabled --pass-access-token, this will pass the token to the backend
      auth_request_set $token  $upstream_http_x_auth_request_access_token;
      proxy_set_header X-Access-Token $token;

      proxy_pass <http://prefect_ui>;
    }

    # The /api/ location is for the API, also protected by the proxy
    location /wizardous/ {
      auth_request /oauth2/auth;
      error_page 401 = /oauth2/sign_in;

      auth_request_set $user  $upstream_http_x_auth_request_user;
      auth_request_set $email $upstream_http_x_auth_request_email;
      proxy_set_header X-User $user;
      proxy_set_header X-Email $email;

      proxy_pass <http://api>;
    }
    

    # Skip favicon.ico
    location = /favicon.ico {
      return 204;
    }
  }
}
m
✅ thought for 166.9 seconds
Good news: oauth2-proxy is authenticating successfully now. The 502 is coming from NGINX when it tries to proxy to an upstream after auth. Based on your logs, oauth2-proxy finishes the callback with a 302 to “/”, then NGINX does: - subrequest to /oauth2/auth (should be 200) - proxy_pass to upstream “prefect_ui” (server:4200) A 502 here almost always means NGINX can’t reach the upstream (connection refused/timeout) or the upstream closed the connection early. Let’s fix likely causes and verify connectivity. Do these three things: 1) Verify connectivity from NGINX to Prefect Server - From your host: - docker compose logs -f nginx | grep -i upstream - docker compose exec nginx sh -lc 'apk add --no-cache curl || apt-get update && apt-get install -y curl || true; curl -sv http://server:4200/ | head; curl -sv http://server:4200/api/health' - If curl to server:4200 fails from the nginx container, the 502 is explained (DNS/network). If it succeeds, proceed to step 2. 2) Harden NGINX proxying (headers + HTTP/1.1 + websockets/SSE) Replace your server block with the below (keeps your routes but adds the critical proxy headers, HTTP/1.1, and upgrade handling). This also sets longer timeouts for API streaming. ``` map $http_upgrade $connection_upgrade { default upgrade; '' close; } upstream prefect_ui { server server:4200; } upstream oauth2_proxy { server oauth2-proxy:4180; } upstream api { server api:9832; } server { listen 80; server_name nginx; # OAuth2 Proxy endpoints location /oauth2/ { proxy_pass http://oauth2_proxy; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Uri $request_uri; proxy_set_header X-Auth-Request-Redirect $request_uri; } location = /oauth2/auth { proxy_pass http://oauth2_proxy; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Uri $request_uri; proxy_set_header Content-Length ""; proxy_pass_request_body off; proxy_set_header X-Auth-Request-Redirect $request_uri; } # Prefect UI and API (same upstream) location / { auth_request /oauth2/auth; error_page 401 = /oauth2/sign_in; # optional: surface user info to upstream auth_request_set $user $upstream_http_x_auth_request_user; auth_request_set $email $upstream_http_x_auth_request_email; proxy_set_header X-User $user; proxy_set_header X-Email $email; auth_request_set $token $upstream_http_x_auth_request_access_token; proxy_set_header X-Access-Token $token; proxy_pass http://prefect_ui; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_read_timeout 300s; proxy_send_timeout 300s; } # Your custom API (still protected) location /wizardous/ { auth_request /oauth2/auth; error_page 401 = /oauth2/sign_in; auth_request_set $user $upstream_http_x_auth_request_user; auth_request_set $email $upstream_http_x_auth_request_email; proxy_set_header X-User $user; proxy_set_header X-Email $email; proxy_pass http://api; proxy_http_version 1.1; proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; } location = /favicon.ico { return 204; } }
Copy code
3) Fix Prefect Server UI’s API URL
Right now your server has PREFECT_UI_API_URL="<http://localhost:4200/api>" which is wrong for traffic coming via NGINX “nginx”. Change it to the external URL the browser actually uses. Since your browser container reaches the site at <http://nginx>:

- In the server service:
  - PREFECT_UI_API_URL=<http://nginx/api>

Then restart the stack.

Also add in oauth2-proxy (for auth_request headers you’re reading in nginx):
set_xauthrequest = true pass_authorization_header = true ``` If you still see 502: - Share the NGINX error lines (they’re very specific), e.g. “connect() failed (111: Connection refused) while connecting to upstream” or “upstream prematurely closed connection”. - From the nginx container, show the outputs of: - getent hosts server - printf 'GET /api/health HTTP/1.1\r\nHost: server\r\n\r\n' | nc -v server 4200 Once we get a clean 200 from / and /api/health through NGINX, we can finalize the UI/API URLs and then harden cookies and group-based access. References: - Example Prefect behind reverse proxy (with websockets/SSE): Example: Prefect + reverse proxy - oauth2-proxy config overview: oauth2-proxy configuration
m
@Marvin Nginx logs:
Copy code
172.20.0.10 - - [30/Aug/2025:23:17:17 +0000] "GET /oauth2/callback?state=6EnzKVrJ2flouQolVuBT4pdgl_1MmU7o1IjFnm2vwHY%3A%2Foauth2%2Fcallback?&session_state=1e14678d-c3f7-4311-87cf-955bad92ce45&iss=http%3A%2F%2Fkeycloak%3A8080%2Frealms%2Fwizard&code=79de3057-346e-44c4-bd15-483ab30ab89c.1e14678d-c3f7-4311-87cf-955bad92ce45.e954fcbe-a31c-41ee-8c6b-da7f80aafa50 HTTP/1.1" 502 157 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0"

2025/08/30 23:17:17 [error] 22#22: *26 upstream sent too big header while reading response header from upstream, client: 172.20.0.10, server: nginx, request: "GET /oauth2/callback?state=6EnzKVrJ2flouQolVuBT4pdgl_1MmU7o1IjFnm2vwHY%3A%2Foauth2%2Fcallback?&session_state=1e14678d-c3f7-4311-87cf-955bad92ce45&iss=http%3A%2F%2Fkeycloak%3A8080%2Frealms%2Fwizard&code=79de3057-346e-44c4-bd15-483ab30ab89c.1e14678d-c3f7-4311-87cf-955bad92ce45.e954fcbe-a31c-41ee-8c6b-da7f80aafa50 HTTP/1.1", upstream: "<http://172.20.0.8:4180/oauth2/callback?state=6EnzKVrJ2flouQolVuBT4pdgl_1MmU7o1IjFnm2vwHY%3A%2Foauth2%2Fcallback?&session_state=1e14678d-c3f7-4311-87cf-955bad92ce45&iss=http%3A%2F%2Fkeycloak%3A8080%2Frealms%2Fwizard&code=79de3057-346e-44c4-bd15-483ab30ab89c.1e14678d-c3f7-4311-87cf-955bad92ce45.e954fcbe-a31c-41ee-8c6b-da7f80aafa50>", host: "nginx"

172.20.0.10 - - [30/Aug/2025:23:17:17 +0000] "GET /favicon.ico HTTP/1.1" 204 0 "<http://nginx/oauth2/callback?state=6EnzKVrJ2flouQolVuBT4pdgl_1MmU7o1IjFnm2vwHY%3A%2Foauth2%2Fcallback?&session_state=1e14678d-c3f7-4311-87cf-955bad92ce45&iss=http%3A%2F%2Fkeycloak%3A8080%2Frealms%2Fwizard&code=79de3057-346e-44c4-bd15-483ab30ab89c.1e14678d-c3f7-4311-87cf-955bad92ce45.e954fcbe-a31c-41ee-8c6b-da7f80aafa50>" "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0"

172.20.0.10 - - [30/Aug/2025:23:22:04 +0000] "GET /favicon.ico HTTP/1.1" 204 0 "<http://nginx/oauth2/callback?state=6EnzKVrJ2flouQolVuBT4pdgl_1MmU7o1IjFnm2vwHY%3A%2Foauth2%2Fcallback?&session_state=1e14678d-c3f7-4311-87cf-955bad92ce45&iss=http%3A%2F%2Fkeycloak%3A8080%2Frealms%2Fwizard&code=79de3057-346e-44c4-bd15-483ab30ab89c.1e14678d-c3f7-4311-87cf-955bad92ce45.e954fcbe-a31c-41ee-8c6b-da7f80aafa50>" "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0"

172.20.0.10 - - [30/Aug/2025:23:22:05 +0000] "GET /oauth2/callback? HTTP/1.1" 500 2815 "<http://nginx/oauth2/callback?state=K3JplFChZU1FUmKwVJvtvPrnwNi0ffkPsenrFm8_Urc%3A%2F&session_state=1e14678d-c3f7-4311-87cf-955bad92ce45&iss=http%3A%2F%2Fkeycloak%3A8080%2Frealms%2Fwizard&code=f94e23eb-83ae-4346-945b-f3c22f965b63.1e14678d-c3f7-4311-87cf-955bad92ce45.e954fcbe-a31c-41ee-8c6b-da7f80aafa50>" "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0"

172.20.0.10 - - [30/Aug/2025:23:22:05 +0000] "GET /oauth2/static/css/bulma.min.css HTTP/1.1" 200 207302 "<http://nginx/oauth2/callback>?" "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0"

172.20.0.10 - - [30/Aug/2025:23:22:05 +0000] "GET /oauth2/static/css/all.min.css HTTP/1.1" 200 102025 "<http://nginx/oauth2/callback>?" "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0"

172.20.0.10 - - [30/Aug/2025:23:22:05 +0000] "GET /favicon.ico HTTP/1.1" 204 0 "<http://nginx/oauth2/callback>?" "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0"

172.20.0.10 - - [30/Aug/2025:23:22:06 +0000] "GET /oauth2/static/webfonts/fa-solid-900.woff2 HTTP/1.1" 200 149908 "<http://nginx/oauth2/static/css/all.min.css>" "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0"
m
✅ thought for 42.4 seconds
Boom — that 502 pinpoints the issue: upstream sent too big header while reading response header from upstream That’s NGINX saying the response headers from oauth2-proxy (almost always big Set-Cookie headers with OIDC session data) exceed its default proxy buffer size. Two ways to fix it: A) Quick fix: increase NGINX proxy buffer sizes for /oauth2/* - Apply these directives (server or http block; I like scoping them to /oauth2/*):
Copy code
http {
  # ... existing ...

  # Bigger buffers for large Set-Cookie headers from oauth2-proxy
  proxy_buffer_size 64k;
  proxy_buffers 16 64k;
  proxy_busy_buffers_size 128k;

  server {
    listen 80;
    server_name nginx;

    location /oauth2/ {
      proxy_pass <http://oauth2_proxy>;
      proxy_http_version 1.1;
      proxy_set_header Host               $host;
      proxy_set_header X-Real-IP          $remote_addr;
      proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto  $scheme;
      proxy_set_header X-Forwarded-Host   $host;
      proxy_set_header X-Forwarded-Uri    $request_uri;
      proxy_set_header X-Auth-Request-Redirect $request_uri;

      # Important: buffer sizes for big Set-Cookie
      proxy_buffer_size 64k;
      proxy_buffers 16 64k;
      proxy_busy_buffers_size 128k;
    }

    location = /oauth2/auth {
      proxy_pass <http://oauth2_proxy>;
      proxy_http_version 1.1;
      proxy_set_header Host               $host;
      proxy_set_header X-Real-IP          $remote_addr;
      proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto  $scheme;
      proxy_set_header X-Forwarded-Host   $host;
      proxy_set_header X-Forwarded-Uri    $request_uri;
      proxy_set_header Content-Length     "";
      proxy_pass_request_body off;

      # Same buffering
      proxy_buffer_size 64k;
      proxy_buffers 16 64k;
      proxy_busy_buffers_size 128k;
    }

    # keep your other locations as before (add Upgrade/Connection if you haven’t yet)
  }
}
B) Better long-term: use oauth2-proxy Redis session store (small cookies) - Instead of stuffing tokens into a cookie, store session server-side in Redis. You already run Redis; use a separate DB index to avoid conflicts with Prefect (e.g., DB 2). Add to oauth2-proxy.cfg:
Copy code
# Keep what you have, plus:
session_store_type = "redis"
redis_connection_url = "<redis://redis:6379/2>"  # use a different DB than Prefect (Prefect uses 0)
# optional tuning
cookie_expire = "8h"
cookie_refresh = "1h"
Then restart oauth2-proxy. This typically shrinks Set-Cookie drastically and avoids having to inflate buffers. Also double-check these from earlier: - oauth2-proxy.cfg should include
Copy code
cookie_secure = false
cookie_samesite = "lax"
set_xauthrequest = true
pass_authorization_header = true
- Your Prefect server env should point the UI to the API via NGINX, not localhost:
Copy code
PREFECT_UI_API_URL=<http://nginx/api>
(You currently have http://localhost:4200/api; change it and restart the server.) Validate quickly: - From the nginx container: - curl -I http://oauth2-proxy:4180/oauth2/sign_in - curl -I http://server:4200/api/health - Try login again; the /oauth2/callback should 302 to “/” without 502. If you still see errors, please share: - Any new NGINX error lines after the buffer changes - The response headers size from oauth2-proxy (curl -sv http://oauth2-proxy:4180/oauth2/sign_in | sed -n '1,/^$/p') to confirm header magnitude Docs for reference: - NGINX proxy buffering: proxy_buffer_size and proxy_buffers - oauth2-proxy session stores: Session storage
m
@Marvin Given this nginx:
Copy code
# See reference:
# <https://github.com/pasha-r/oauth2_proxy/blob/master/docs/configuration/configuration.md#config-file>

events {
  worker_connections 1024;
}

http {
  # Bigger buffers for large Set-Cookie headers from oauth2-proxy
  proxy_buffer_size 64k;
  proxy_buffers 16 64k;
  proxy_busy_buffers_size 128k;

  upstream prefect_ui {
    server server:4200;
  }
  upstream oauth2_proxy {
    server oauth2-proxy:4180;
  }
  upstream api {
    server api:9832;
  }

  server {
    listen 80;
    server_name nginx;

    # The /oauth2/ location handles all OAuth2 Proxy traffic
    location /oauth2/ {
      proxy_pass <http://oauth2_proxy>;
      proxy_http_version 1.1;
      proxy_set_header Host               $host;
      proxy_set_header X-Real-IP          $remote_addr;
      proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto  $scheme;
      proxy_set_header X-Forwarded-Host   $host;
      proxy_set_header X-Forwarded-Uri    $request_uri;
      proxy_set_header X-Auth-Request-Redirect $request_uri;

      # Important: buffer sizes for big Set-Cookie
      proxy_buffer_size 64k;
      proxy_buffers 16 64k;
      proxy_busy_buffers_size 128k;
    }

    # The /oauth2/auth location is the actual authentication endpoint
    location = /oauth2/auth {
      proxy_pass <http://oauth2_proxy>;
      proxy_http_version 1.1;
      proxy_set_header Host               $host;
      proxy_set_header X-Real-IP          $remote_addr;
      proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto  $scheme;
      proxy_set_header X-Forwarded-Host   $host;
      proxy_set_header X-Forwarded-Uri    $request_uri;
      proxy_set_header Content-Length     "";
      proxy_pass_request_body off;

      # Same buffering
      proxy_buffer_size 64k;
      proxy_buffers 16 64k;
      proxy_busy_buffers_size 128k;
    }

    # The / location is for the main Prefect UI
    location / {
      auth_request /oauth2/auth;
      error_page 401 = /oauth2/sign_in;

      auth_request_set $user  $upstream_http_x_auth_request_user;
      auth_request_set $email $upstream_http_x_auth_request_email;
      proxy_set_header X-User $user;
      proxy_set_header X-Email $email;

      # if you enabled --pass-access-token, this will pass the token to the backend
      auth_request_set $token  $upstream_http_x_auth_request_access_token;
      proxy_set_header X-Access-Token $token;

      proxy_pass <http://prefect_ui>;
    }

    # The /api/ location is for the API, also protected by the proxy
    location /wizardous/ {
      auth_request /oauth2/auth;
      error_page 401 = /oauth2/sign_in;

      auth_request_set $user  $upstream_http_x_auth_request_user;
      auth_request_set $email $upstream_http_x_auth_request_email;
      proxy_set_header X-User $user;
      proxy_set_header X-Email $email;

      rewrite ^/wizardous/(.*) /$1 break;

      proxy_pass <http://api>;
    }
    

    # # Skip favicon.ico
    # location = /favicon.ico {
    #   return 204;
    # }
  }
}
/wizardous/* routes to an API, so it shouldn't be accessed via a login page, it should use tokens instead, how do I make that happen and how do I generate tokens per user
m
✅ thought for 96.0 seconds
Great question. You can keep oauth2-proxy for the UI, while making your API at /wizardous/ accept Bearer tokens and never redirect to a login page. Two patterns you can choose from: A) Use oauth2-proxy only to validate and inject tokens for browser calls - API calls from an already-authenticated browser session get a Bearer token injected by NGINX (taken from oauth2-proxy). - If unauthenticated, API returns 401 JSON (no redirect). - External clients can also send Authorization: Bearer <token> and you can choose to validate that either via oauth2-proxy or in your API. B) Bypass oauth2-proxy for /wizardous/ and validate JWTs directly in your API - Straight proxy in NGINX, your API validates tokens using Keycloak’s JWKS. - Good if you want your API to be self-contained. Here’s how to do A (recommended if your UI and API are on the same origin): 1) Update oauth2-proxy config to surface tokens/headers Make sure you have these:
Copy code
set_xauthrequest = true
pass_authorization_header = true
pass_access_token = true
2) Ensure /oauth2/auth gets the Authorization header Add this to your NGINX /oauth2/auth location so Bearer tokens sent by clients reach oauth2-proxy:
Copy code
proxy_set_header Authorization $http_authorization;
3) Make /wizardous/ return 401 (no redirect) and inject Bearer to your API Replace your /wizardous/ block with:
Copy code
location /wizardous/ {
  # Validate via oauth2-proxy, but DO NOT redirect to login
  auth_request /oauth2/auth;

  # If not authenticated, return a JSON 401 instead of redirecting
  error_page 401 = @api_unauthorized;

  # Optional: forward user info
  auth_request_set $user   $upstream_http_x_auth_request_user;
  auth_request_set $email  $upstream_http_x_auth_request_email;
  proxy_set_header X-User  $user;
  proxy_set_header X-Email $email;

  # If authenticated with oauth2-proxy, inject a Bearer token for your API
  auth_request_set $token  $upstream_http_x_auth_request_access_token;
  proxy_set_header Authorization "Bearer $token";

  # Preserve Authorization if client already sent its own token
  if ($http_authorization) {
    proxy_set_header Authorization $http_authorization;
  }

  rewrite ^/wizardous/(.*)$ /$1 break;
  proxy_pass <http://api>;
}

location @api_unauthorized {
  add_header WWW-Authenticate 'Bearer realm="wizard", error="invalid_token"' always;
  default_type application/json;
  return 401 '{"error":"unauthorized"}';
}
Notes: - For browser-originating calls, oauth2-proxy issues the access token; NGINX injects it so your API sees Authorization: Bearer .... - For programmatic clients, they can call /wizardous/* with their own Bearer token; with the 
proxy_set_header Authorization $http_authorization;
line, that token is forwarded to the API. - If you want oauth2-proxy to accept and validate inbound Bearer tokens itself (so only valid tokens get through), you can enable its JWT validation mode (commonly via “skip_jwt_bearer_tokens” and issuer/audience settings). If you prefer, let your API validate instead (see option B below). Option B: Let your API validate JWTs and skip oauth2-proxy for /wizardous/ Use this simpler NGINX block and perform JWT validation in your API:
Copy code
location /wizardous/ {
  # No oauth2_proxy here
  proxy_set_header Authorization $http_authorization;
  rewrite ^/wizardous/(.*)$ /$1 break;
  proxy_pass <http://api>;
}
In your API, validate tokens against Keycloak: - Issuer: http://keycloak:8080/realms/wizard - JWKS: http://keycloak:8080/realms/wizard/protocol/openid-connect/certs - Validate aud to include your API client_id, and exp/nbf, etc. How to get “tokens per user” - Browser users (your UI): They authenticate via oauth2-proxy (Auth Code flow). You don’t need to hand tokens to the browser app if you use the injection method above; NGINX injects the Bearer for the API calls. If you do need the token in JS, you’ll need a different architecture (SPA with PKCE) that talks directly to Keycloak. - CLI or scripts (user tokens):
- Device Authorization Grant (good UX for CLIs) 1) Start device flow
Copy code
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/device/auth>" \
      -d "client_id=wizard-cli" \
      -d "scope=openid profile email offline_access"
2) User visits the verification_uri_complete from the response and logs in 3) Poll for tokens
Copy code
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/token>" \
      -d "grant_type=urn:ietf:params:oauth:grant-type:device_code" \
      -d "client_id=wizard-cli" \
      -d "device_code=...from step 1..."
- Resource Owner Password (Direct Access Grants) — enable on the client; not recommended for prod
Copy code
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/token>" \
      -H "Content-Type: application/x-www-form-urlencoded" \
      -d "client_id=wizard-cli" \
      -d "client_secret=REDACTED" \
      -d "grant_type=password" \
      -d "username=test@test.com" \
      -d "password=YOUR_PASSWORD"
- Service-to-service (no user): Client Credentials grant
Copy code
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/token>" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "client_id=wizard-api" \
    -d "client_secret=REDACTED" \
    -d "grant_type=client_credentials"
Configure clients in Keycloak accordingly (public + PKCE for SPAs, confidential for CLI/machine clients). Make sure the audience of issued tokens includes your API’s client_id (you already added an audience mapper earlier). Docs you may find helpful: - oauth2-proxy + NGINX auth_request and headers: oauth2-proxy: NGINX auth_request - oauth2-proxy session and headers: oauth2-proxy configuration - Keycloak device flow: Keycloak: Device Authorization Grant If you tell me whether you want oauth2-proxy to validate inbound Bearer tokens or you prefer to validate inside your API, I’ll tailor the exact config (including issuer/audience options) and provide a quick token verification snippet for your API stack.
m
@Marvin Now, let's recap I want the API to NOT handle any authentication, rather, the user who requests it, adds the necessary tokens, IDs in the request, then it gets authenticated, and request gets routed to the API
m
✅ thought for 82.7 seconds
Perfect — you want the edge to do all auth, and your API to stay ignorant. The clean pattern is: - Clients call /wizardous/* with Authorization: Bearer <JWT-from-Keycloak> - NGINX uses auth_request to ask oauth2-proxy to validate the token - If valid: request is routed to your API as-is - If missing/invalid: NGINX returns 401 JSON (no redirects, no login page) Here’s exactly how to wire it. 1) oauth2-proxy: enable bearer-token validation (no redirects for API) Add these to your existing oauth2-proxy.cfg:
Copy code
# Keep your OIDC settings for UI login flows
provider = "keycloak-oidc"
oidc_issuer_url = "<http://keycloak:8080/realms/wizard>"
client_id = "wizard-oauth"
client_secret = "REDACTED"
redirect_url = "<http://nginx/oauth2/callback>"
reverse_proxy = true

# Session/cookie settings (for UI)
cookie_secret = "REDACTED"
cookie_secure = false
cookie_samesite = "lax"
set_xauthrequest = true
pass_authorization_header = true

# IMPORTANT: accept & validate incoming JWT bearer tokens for the API
skip_jwt_bearer_tokens = true
extra_jwt_issuers = [ "<http://keycloak:8080/realms/wizard=wizard-api>" ]

# Optional (handy for CORS preflight to the API)
skip_auth_preflight = true
Notes: - extra_jwt_issuers tells oauth2-proxy to validate JWTs whose iss matches your realm and whose aud includes wizard-api. - You’ll need tokens whose aud contains wizard-api. See Keycloak steps below. 2) NGINX: protect /wizardous/ with auth_request (no redirect), pass through Authorization Replace your /wizardous/ block with:
Copy code
location /wizardous/ {
  # Validate via oauth2-proxy using the client's Bearer token
  auth_request /oauth2/auth;

  # Send the Authorization header to oauth2-proxy for validation
  proxy_set_header Authorization $http_authorization;

  # On auth failure, return 401 JSON instead of redirecting to login
  error_page 401 = @api_unauthorized;

  # Optional: forward some identity to your API (not required)
  auth_request_set $user  $upstream_http_x_auth_request_user;
  auth_request_set $email $upstream_http_x_auth_request_email;
  proxy_set_header X-User  $user;
  proxy_set_header X-Email $email;

  # Forward the same Authorization header to your API untouched
  proxy_set_header Authorization $http_authorization;

  # Keep your path rewrite
  rewrite ^/wizardous/(.*)$ /$1 break;

  proxy_pass <http://api>;
  proxy_http_version 1.1;
  proxy_set_header Host              $host;
  proxy_set_header X-Real-IP         $remote_addr;
  proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-Forwarded-Host  $host;
}

location = /oauth2/auth {
  proxy_pass <http://oauth2_proxy>;
  proxy_http_version 1.1;
  proxy_set_header Host               $host;
  proxy_set_header X-Real-IP          $remote_addr;
  proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto  $scheme;
  proxy_set_header X-Forwarded-Host   $host;
  proxy_set_header X-Forwarded-Uri    $request_uri;
  proxy_set_header Content-Length     "";
  proxy_pass_request_body off;

  # CRITICAL: forward inbound Authorization header to oauth2-proxy for validation
  proxy_set_header Authorization $http_authorization;
}

location @api_unauthorized {
  add_header WWW-Authenticate 'Bearer realm="wizard", error="invalid_token"' always;
  default_type application/json;
  return 401 '{"error":"unauthorized"}';
}
- Result: If a request lacks a valid Bearer JWT, oauth2-proxy returns 401 to the subrequest and NGINX returns the JSON 401. No login redirect. 3) Keycloak: issue tokens your edge will accept Create a separate “API” client (recommended): - Client: wizard-api (confidential) - Access type: Confidential (with client secret) - Service Accounts: enable if you want client-credentials tokens - Add Audience mapper so aud includes wizard-api: - Clients > wizard-api > Client scopes or Mappers > Create > Mapper type: Audience - Included Client Audience: wizard-api
- Add to access token: ON Ways clients can get tokens: - Service-to-service (client credentials)
Copy code
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/token>" \
  -d "grant_type=client_credentials" \
  -d "client_id=wizard-api" \
  -d "client_secret=REDACTED"
- Per-user tokens for CLI/automation (Device Authorization Grant)
Copy code
# 1) Start device flow
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/device/auth>" \
  -d "client_id=wizard-api" \
  -d "scope=openid profile email offline_access"
# Follow verification_uri_complete, then:
# 2) Poll for token
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/token>" \
  -d "grant_type=urn:ietf:params:oauth:grant-type:device_code" \
  -d "client_id=wizard-api" \
  -d "device_code=...from step 1..."
- Password grant (Direct Access Grants) for dev only
Copy code
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/token>" \
  -d "grant_type=password" \
  -d "client_id=wizard-api" \
  -d "client_secret=REDACTED" \
  -d "username=test@test.com" \
  -d "password=REDACTED"
4) Test the edge-only auth - No token: - curl -i http://nginx/wizardous/anything - Expect: HTTP/1.1 401 with JSON body {"error":"unauthorized"} - Bad token: - curl -i -H "Authorization: Bearer xyz" http://nginx/wizardous/anything - Expect: 401 - Valid token (aud contains wizard-api): - TOKEN=$(curl -s ... | jq -r .access_token) - curl -i -H "Authorization: Bearer $TOKEN" http://nginx/wizardous/anything - Expect: 200 from your API Optional hardening - Restrict accepted audiences to wizard-api only (as configured) - If your API doesn’t need to see the token, you can strip it and only pass identity headers (X-User/X-Email). If it does, keep forwarding Authorization as shown. - For CORS: with skip_auth_preflight=true, OPTIONS preflights will pass; add Access-Control-* headers in NGINX or your API if you’re calling from browsers. If you want me to tailor the Keycloak client settings (confidential vs public with PKCE, token lifetimes, groups-to-claims) or to add group-based enforcement at NGINX (e.g., only allow tokens with a specific claim), say the word and I’ll provide the exact mappers and NGINX rules.
m
@Marvin
Copy code
# curl -i  <http://nginx/wizardous/health> -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJxTC1ESGpnUWQyd2FLcHZZaEdRZjBTaU5YOXVNTGNQNmNNWEgwTG8tZ0FrIn0.eyJleHAiOjE3NTY2MDExOTMsImlhdCI6MTc1NjYwMDg5MywianRpIjoiZjA5ZTA2MTctNDdlZS00NDk3LTk1ODEtNTExYTcyMzhhMmRhIiwiaXNzIjoiaHR0cDovL2tleWNsb2FrOjgwODAvcmVhbG1zL3dpemFyZCIsImF1ZCI6WyJ3aXphcmQtYXBpIiwiYWNjb3VudCJdLCJzdWIiOiJiYmJmNTU2Zi0wMWMxLTRmNjMtYmY0ZC00M2NmY2Q2MGExYjYiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJ3aXphcmQtYXBpIiwiYWNyIjoiMSIsImFsbG93ZWQtb3JpZ2lucyI6WyJodHRwOi8vbmdpbngiXSwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbImRlZmF1bHQtcm9sZXMtd2l6YXJkIiwib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiJdfSwicmVzb3VyY2VfYWNjZXNzIjp7ImFjY291bnQiOnsicm9sZXMiOlsibWFuYWdlLWFjY291bnQiLCJtYW5hZ2UtYWNjb3VudC1saW5rcyIsInZpZXctcHJvZmlsZSJdfX0sInNjb3BlIjoicHJvZmlsZSBlbWFpbCIsImNsaWVudEhvc3QiOiIxNzIuMjAuMC42IiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJzZXJ2aWNlLWFjY291bnQtd2l6YXJkLWFwaSIsImNsaWVudEFkZHJlc3MiOiIxNzIuMjAuMC42IiwiY2xpZW50X2lkIjoid2l6YXJkLWFwaSJ9.BhznASo2uubvFdgrOhPwu8Z4O2hViMV07jr7vdhFxNjibxKmk_9o7XgPkz7AfLtv5nBlVsBXmUBI4l2LdczPfTVkk210AIIgMm0y5u0PYqGM6EcPmU2a2Trh8uXF3-OZNLqa2YrwGxVaTTIyrMGBfxpcrw9aEC2gEPR-8wYFclnv7xbBEdKTEkwvSPWxyt6sn9y4RyiSlN2ZW9hfFp7HHrXobxxOPq6468wbPeOnZ6g4wjxASQ7T292BwCaIaRwhHHHjfIX9WCsoIXPS7twoRQTfOjSVE_DdjElzin-32js8Mt0cT7u7HPuQcZhk6lJTs0m1T0V8BLsnTJeQsSnx3A"
HTTP/1.1 302 Found
Server: nginx/1.29.1
Date: Sun, 31 Aug 2025 00:44:34 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 382
Connection: keep-alive
Cache-Control: no-cache, no-store, must-revalidate, max-age=0
Expires: Thu, 01 Jan 1970 00:00:00 UTC
Location: <http://keycloak:8080/realms/wizard/protocol/openid-connect/auth?approval_prompt=force&client_id=wizard-oauth&nonce=ycSfxjoWjEuXEpTP3Pbi92JHoQSISohcud_ePhTRqHc&redirect_uri=http%3A%2F%2Fnginx%3A80%2Foauth2%2Fcallback&response_type=code&scope=openid+email+profile&state=DL03Zzbb66tZVYQJNNHtKRGsL_tMQYCw5DFgtF4-roo%3A%2Fwizardous%2Fhealth>
Set-Cookie: _oauth2_proxy_csrf=ZTKV3QqJaZuMbAgZo4srLCPO9D1p7ifRUXl_VpoiASJN__19jBXdAQFRxLVHQWW4TYjr7wKBG6jGz0_9ALVSaW1lkqfLgt1SN51IPGwRRGmFEBiE53Z6GcM=|1756601074|T2DkBNZ9_fV2t0Zjvw3fU2dmjffKx3-esz1KlUQQ6vY=; Path=/; Expires=Sun, 31 Aug 2025 00:59:34 GMT; HttpOnly; SameSite=Lax

<a href="<http://keycloak:8080/realms/wizard/protocol/openid-connect/auth?approval_prompt=force&client_id=wizard-oauth&nonce=ycSfxjoWjEuXEpTP3Pbi92JHoQSISohcud_ePhTRqHc&redirect_uri=http%3A%2F%2Fnginx%3A80%2Foauth2%2Fcallback&response_type=code&scope=openid+email+profile&state=DL03Zzbb66tZVYQJNNHtKRGsL_tMQYCw5DFgtF4-roo%3A%2Fwizardous%2Fhealth>">Found</a>.
m
✅ thought for 106.2 seconds
You’re getting the 302 because your /wizardous/ block is still “browser-style” auth. Two things are happening: - The Authorization header is not being forwarded to the auth subrequest, so oauth2-proxy doesn’t see your Bearer token. - Your API block maps 401 to /oauth2/sign_in, which triggers a browser login redirect (302 to Keycloak). Here’s the exact fix to make /wizardous/ accept Bearer tokens and never redirect: 1) Update oauth2-proxy.cfg to validate inbound JWTs - This tells oauth2-proxy to accept and verify Bearer tokens (JWT) from Keycloak for the API, instead of starting a login flow.
Copy code
# keep your existing OIDC settings (issuer, client_id, etc.)

# Accept & validate incoming JWT bearer tokens
skip_jwt_bearer_tokens = true
extra_jwt_issuers = [ "<http://keycloak:8080/realms/wizard=wizard-api>" ]

# Nice-to-have for browser CORS preflights
skip_auth_preflight = true

# Ensure these are present for UI + nginx auth_request integration
set_xauthrequest = true
pass_authorization_header = true
pass_access_token = true
Notes: - extra_jwt_issuers format is "ISSUER=AUDIENCE". Your sample token’s iss is http://keycloak:8080/realms/wizard and aud includes wizard-api, so this matches. - Docs: oauth2-proxy config and skip_jwt_bearer_tokens 2) Update NGINX to pass Authorization to oauth2-proxy and return 401 JSON for API - Replace your /wizardous/ and /oauth2/auth blocks with the following:
Copy code
# Subrequest endpoint used by auth_request
location = /oauth2/auth {
  proxy_pass <http://oauth2_proxy>;
  proxy_http_version 1.1;

  proxy_set_header Host               $host;
  proxy_set_header X-Real-IP          $remote_addr;
  proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto  $scheme;
  proxy_set_header X-Forwarded-Host   $host;
  proxy_set_header X-Forwarded-Uri    $request_uri;

  # CRITICAL: forward Authorization header so oauth2-proxy can validate Bearer JWTs
  proxy_set_header Authorization      $http_authorization;

  proxy_set_header Content-Length     "";
  proxy_pass_request_body off;
}

# Your API — validate token at the edge, never redirect
location /wizardous/ {
  auth_request /oauth2/auth;

  # On auth failure, return JSON 401 (no login redirect)
  error_page 401 = @api_unauthorized;

  # Forward identity (optional)
  auth_request_set $user  $upstream_http_x_auth_request_user;
  auth_request_set $email $upstream_http_x_auth_request_email;
  proxy_set_header X-User  $user;
  proxy_set_header X-Email $email;

  # Forward the same Authorization header to your API (or drop it if you don’t want API to see it)
  proxy_set_header Authorization $http_authorization;

  # Keep your path rewrite
  rewrite ^/wizardous/(.*)$ /$1 break;

  proxy_pass <http://api>;
  proxy_http_version 1.1;
  proxy_set_header Host               $host;
  proxy_set_header X-Real-IP          $remote_addr;
  proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto  $scheme;
  proxy_set_header X-Forwarded-Host   $host;
}

location @api_unauthorized {
  add_header WWW-Authenticate 'Bearer realm="wizard", error="invalid_token"' always;
  default_type application/json;
  return 401 '{"error":"unauthorized"}';
}
Key points: - We removed error_page 401 = /oauth2/sign_in from the API block. That was causing your redirect. - We forward Authorization to the auth subrequest; without this, oauth2-proxy won’t see the Bearer token and will try to start a login. 3) Ensure tokens your clients use have the right audience - Your posted token has aud ["wizard-api","account"] and iss http://keycloak:8080/realms/wizard — perfect for the extra_jwt_issuers setting above. - In Keycloak, make sure the “wizard-api” client has an Audience mapper that adds wizard-api to the access token’s aud.
- How to mint tokens (examples): - Client credentials (service-to-service):
Copy code
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/token>" \
      -d "grant_type=client_credentials" \
      -d "client_id=wizard-api" \
      -d "client_secret=REDACTED"
- Device flow (per-user, good for CLIs):
Copy code
# Start device flow
    curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/device/auth>" \
      -d "client_id=wizard-api" \
      -d "scope=openid profile email offline_access"

    # Poll for token after user verifies
    curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/token>" \
      -d "grant_type=urn:ietf:params:oauth:grant-type:device_code" \
      -d "client_id=wizard-api" \
      -d "device_code=...from previous step..."
- Direct Access Grants (dev only):
Copy code
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/token>" \
      -d "grant_type=password" \
      -d "client_id=wizard-api" \
      -d "client_secret=REDACTED" \
      -d "username=test@test.com" \
      -d "password=REDACTED"
4) Test it - No token: - curl -i http://nginx/wizardous/health - Expect 401 with JSON {"error":"unauthorized"} - Bad token: - curl -i -H "Authorization: Bearer xyz" http://nginx/wizardous/health - Expect 401 - Valid token (aud includes wizard-api): - TOKEN=$(curl -s ... | jq -r .access_token) - curl -i -H "Authorization: Bearer $TOKEN" http://nginx/wizardous/health - Expect 200 from your API Optional: - If you need CORS for the API from a browser app, enable skip_auth_preflight (above) and add Access-Control-* headers in NGINX or your API responses. - If you want to allow multiple audiences (e.g., wizard-oauth too), add to extra_jwt_issuers:
Copy code
extra_jwt_issuers = [
    "<http://keycloak:8080/realms/wizard=wizard-api>",
    "<http://keycloak:8080/realms/wizard=wizard-oauth>"
  ]
If you apply these changes and still see a redirect, please paste: - The /wizardous/ and /oauth2/auth blocks you’re running - The oauth2-proxy logs for a failing request (it will say whether it saw/validated a JWT) - The response headers for GET -I http://nginx/wizardous/health with and without Authorization so we can verify the 401 path is used correctly.
m
@Marvin How to extend token expiry
m
✅ thought for 47.7 seconds
Short answer: increase lifetimes in Keycloak (access/refresh/SSO/offline), and, for the UI, let oauth2-proxy refresh tokens using offline_refresh. For API bearer tokens, either mint longer-lived access tokens or have clients refresh. Decide what you want to extend - UI browser session via oauth2-proxy - API Bearer tokens used at /wizardous/* A) Extend UI session (oauth2-proxy + Keycloak) 1) Keycloak: allow long-lived refresh (optional but recommended) - Realm Settings → Sessions: - SSO Session Idle / Max: increase as desired (e.g., 8h / 24h) - Offline Session Idle / Max: increase if you’ll use offline_access (e.g., 30d / 90d) - Realm Settings → Tokens: - Access Token Lifespan: e.g., 30m–60m - Refresh Token Lifespan: increase as needed - Refresh Token Max Reuse / Rotation: tune per your security needs Docs: https://www.keycloak.org/docs/latest/server_admin/#_tokens and https://www.keycloak.org/docs/latest/server_admin/#_timeouts 2) oauth2-proxy: request refresh tokens and refresh the session - Ensure you request offline access and refresh periodically - Consider Redis session store to avoid huge cookies Example:
Copy code
# existing OIDC settings...

# Ask Keycloak for refresh tokens
scope = "openid profile email offline_access"

# Session lifetime policy at the edge (bounded by Keycloak refresh lifetimes)
cookie_expire  = "12h"     # overall session cookie lifetime
cookie_refresh = "1h"      # proactively refresh tokens every hour

# Recommended to keep cookies small/persistent
session_store_type   = "redis"
redis_connection_url = "<redis://redis:6379/2>"

# Keep these from earlier
cookie_secure = false
cookie_samesite = "lax"
set_xauthrequest = true
pass_authorization_header = true
pass_access_token = true
B) Extend API Bearer tokens at /wizardous/* Because you’re validating tokens at the edge (oauth2-proxy) and keeping the API unaware: - If you want longer-lived bearer tokens: - In Keycloak, increase Access Token Lifespan for the API client (wizard-api) or at the realm. - Clients → wizard-api → Settings → Advanced (override Access Token Lifespan), or use realm defaults (Realm Settings → Tokens). - Recommended approach: keep access tokens short (e.g., 15–30m) and have clients refresh. - User-based clients: use Device Flow or Password (dev only) to obtain a refresh_token and exchange it for new access tokens when needed. - Service-to-service (client_credentials): typically no refresh token; fetch a new access token when the old one expires. Examples to mint/refresh tokens - Client credentials (service-to-service):
Copy code
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/token>" \
  -d "grant_type=client_credentials" \
  -d "client_id=wizard-api" \
  -d "client_secret=REDACTED"
- Device flow (per-user, CLI-friendly) — yields refresh_token:
Copy code
# Start device flow
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/device/auth>" \
  -d "client_id=wizard-api" \
  -d "scope=openid profile email offline_access"

# After user verifies via verification_uri_complete, poll for tokens:
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/token>" \
  -d "grant_type=urn:ietf:params:oauth:grant-type:device_code" \
  -d "client_id=wizard-api" \
  -d "device_code=...from previous step..."
- Refresh with a refresh_token (for user-based flows):
Copy code
curl -s -X POST "<http://keycloak:8080/realms/wizard/protocol/openid-connect/token>" \
  -d "grant_type=refresh_token" \
  -d "client_id=wizard-api" \
  -d "refresh_token=REDACTED"
Security notes - Prefer short-lived access tokens + refresh over very long-lived access tokens. - If you do extend lifetimes, consider enabling refresh token rotation and setting sensible SSO/Offline session caps. - For the UI, oauth2-proxy can silently refresh in the background using the refresh token, as long as the realm/client lifetimes allow it.
If you tell me your target durations (e.g., UI session 12h, API tokens 1h), I’ll provide precise Keycloak values and the exact oauth2-proxy config to match.
m
@Marvin oauth2-proxy error when trying to authenticate with token:
Copy code
172.20.0.6 - c21f18a2-4153-41fb-98e9-c9643bc039b5 - - [2025/08/31 01:04:54] nginx GET - "/oauth2/auth" HTTP/1.1 "curl/8.14.1" 401 13 0.086

[2025/08/31 01:04:54] [jwt_session.go:51] Error retrieving session from token in Authorization header: [unable to verify bearer token, could not create session from token: audience from claim aud with value [wizard-api account] does not match with any of allowed audiences map[wizard-oauth:{}], email in id_token (bbbf556f-01c1-4f63-bf4d-43cfcd60a1b6) isn't verified]
Here's oauth2 proxy configs:
Copy code
provider = "keycloak-oidc"
oidc_issuer_url = "<http://keycloak:8080/realms/wizard>"
client_id = "wizard-oauth"
client_secret = "olcWi1lGcyrU82MtdvNgK8Nm7x4C8ieA"
email_domains = "*"
insecure_oidc_allow_unverified_email = true
http_address = "0.0.0.0:4180"
redirect_url = "<http://nginx:80/oauth2/callback>"
pass_access_token = true
skip_provider_button = true
reverse_proxy = true

cookie_secret = "bHEyY2E0dEZHSTlnY2hJa0FRWnRERURxaDg4Qm5qY3k="
cookie_secure = false           # IMPORTANT for HTTP; otherwise the cookie won't be sent
cookie_samesite = "lax"

set_xauthrequest = true
pass_authorization_header = true

# IMPORTANT: accept & validate incoming JWT bearer tokens for the API
skip_jwt_bearer_tokens = true
extra_jwt_issuers = [ "<http://keycloak:8080/realms/wizard=wizard-api>" ]

# Optional (handy for CORS preflight to the API)
skip_auth_preflight = true
m
✅ thought for 116.8 seconds
You’re very close — oauth2-proxy is still validating the token against the UI client_id (“wizard-oauth”), which means your extra_jwt_issuers isn’t being applied. That’s why the log shows allowed audiences map[wizard-oauth:{}]. Root cause - Your config has angle brackets around the issuer string. In oauth2-proxy config, values must not include Slack-style <...> wrappers, or they won’t match the token’s iss. Fix the config - Remove angle brackets and keep the exact issuer and audience. - Keep skip_jwt_bearer_tokens = true so oauth2-proxy validates Bearer JWTs instead of starting a login flow. Replace this block in oauth2-proxy.cfg:
Copy code
# Accept & validate incoming JWT bearer tokens for the API
skip_jwt_bearer_tokens = true
extra_jwt_issuers = [ "<http://keycloak:8080/realms/wizard=wizard-api>" ]

# Optional (handy for CORS preflight to the API)
skip_auth_preflight = true
Also ensure these are present (for nginx auth_request + UI):
Copy code
set_xauthrequest = true
pass_authorization_header = true
pass_access_token = true
Double-check NGINX forwards the Authorization header to the auth subrequest:
Copy code
location = /oauth2/auth {
  proxy_pass <http://oauth2_proxy>;
  proxy_set_header Authorization $http_authorization;
  proxy_set_header Content-Length "";
  proxy_pass_request_body off;
  # ...other forwarded headers...
}
Test - Call oauth2-proxy directly with your token (should return 200):
Copy code
curl -i -H "Authorization: Bearer $TOKEN" <http://oauth2-proxy:4180/oauth2/auth>
- Through NGINX to your API path:
Copy code
curl -i -H "Authorization: Bearer $TOKEN" <http://nginx/wizardous/health>
Expected: 200 from your API. If missing/invalid token: 401 JSON from NGINX (no redirect). Notes - The “email isn’t verified” part of the log came from oauth2-proxy trying to build a browser session (ID token path). Once extra_jwt_issuers matches and the Bearer JWT is accepted, that check won’t apply to your API calls. - If you want to accept multiple audiences, list them:
Copy code
extra_jwt_issuers = [
  "<http://keycloak:8080/realms/wizard=wizard-api>",
  "<http://keycloak:8080/realms/wizard=wizard-oauth>"
]
If it still 401s, please paste: - The first few lines of oauth2-proxy startup logs (it will echo parsed config, including extra_jwt_issuers) - The output of curl -i http://oauth2-proxy:4180/oauth2/auth with and without Authorization so we can see the status codes end-to-end.
m
@Marvin Still gives 401 Here's are the startup logs:
Copy code
[2025/08/31 00:49:04] [provider.go:55] Performing OIDC Discovery...

[2025/08/31 00:49:04] [provider.go:55] Performing OIDC Discovery...

[2025/08/31 00:49:04] [providers.go:145] Warning: Your provider supports PKCE methods ["plain" "S256"], but you have not enabled one with --code-challenge-method

[2025/08/31 00:49:04] [oauthproxy.go:156] Skipping JWT tokens from configured OIDC issuer: "<http://keycloak:8080/realms/wizard>"

[2025/08/31 00:49:04] [oauthproxy.go:158] Skipping JWT tokens from extra JWT issuer: "<http://keycloak:8080/realms/wizard=wizard-api>"

[2025/08/31 00:49:04] [oauthproxy.go:166] OAuthProxy configured for Keycloak OIDC Client ID: wizard-oauth

[2025/08/31 00:49:04] [oauthproxy.go:172] Cookie settings: name:_oauth2_proxy secure(https):false httponly:true expiry:168h0m0s domains: path:/ samesite:lax refresh:disabled
m
✅ thought for 58.7 seconds
Your oauth2-proxy is still reading the issuer/audience with angle brackets, so it’s ignoring your extra_jwt_issuers and falling back to the UI client_id (wizard-oauth) — that’s why you still get 401. From your startup logs: - Skipping JWT tokens from configured OIDC issuer: "http://keycloak:8080/realms/wizard" - Skipping JWT tokens from extra JWT issuer: "http://keycloak:8080/realms/wizard=wizard-api" Those < > must not be in the actual config file. Fix the config to use raw values (no angle brackets), then restart the container. Do this exactly: 1) Edit proxies/oauth2-proxy.cfg - Remove all < and > around URLs - Keep the audience mapping exactly as ISSUER=AUDIENCE Working snippet:
Copy code
provider = "keycloak-oidc"
oidc_issuer_url = "<http://keycloak:8080/realms/wizard>"
client_id = "wizard-oauth"
client_secret = "REDACTED"

redirect_url = "<http://nginx/oauth2/callback>"
reverse_proxy = true

email_domains = "*"
insecure_oidc_allow_unverified_email = true
pass_access_token = true

cookie_secret = "REDACTED"
cookie_secure = false
cookie_samesite = "lax"

set_xauthrequest = true
pass_authorization_header = true

# Accept & validate incoming JWT bearer tokens for the API
skip_jwt_bearer_tokens = true
extra_jwt_issuers = [ "<http://keycloak:8080/realms/wizard=wizard-api>" ]

# Optional: lets CORS preflight skip auth
skip_auth_preflight = true
2) Ensure NGINX forwards Authorization to the auth subrequest
Copy code
location = /oauth2/auth {
  proxy_pass <http://oauth2_proxy>;
  proxy_http_version 1.1;

  proxy_set_header Host               $host;
  proxy_set_header X-Real-IP          $remote_addr;
  proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto  $scheme;
  proxy_set_header X-Forwarded-Host   $host;
  proxy_set_header X-Forwarded-Uri    $request_uri;

  # CRITICAL
  proxy_set_header Authorization      $http_authorization;

  proxy_set_header Content-Length     "";
  proxy_pass_request_body off;
}
3) Restart oauth2-proxy and confirm the logs You should see: - Skipping JWT tokens from configured OIDC issuer: http://keycloak:8080/realms/wizard - Skipping JWT tokens from extra JWT issuer: http://keycloak:8080/realms/wizard=wizard-api (no quotes with angle brackets) 4) Test directly against oauth2-proxy, then via NGINX - Direct:
Copy code
curl -i -H "Authorization: Bearer $TOKEN" <http://oauth2-proxy:4180/oauth2/auth>
Expect 200 when $TOKEN’s iss is http://keycloak:8080/realms/wizard and aud includes wizard-api. - Through NGINX to your API:
Copy code
curl -i -H "Authorization: Bearer $TOKEN" <http://nginx/wizardous/health>
Expect 200 from your API. Without/invalid token should be 401 JSON (no redirect). If it still 401s after removing the brackets: - Paste the updated startup logs (first 10 lines) - Paste the output of the direct check: curl -i -H "Authorization: Bearer $TOKEN" http://oauth2-proxy:4180/oauth2/auth Docs for reference: - oauth2-proxy auth_request and JWT settings: https://oauth2-proxy.github.io/oauth2-proxy/docs/configuration/overview
m
@Marvin There ARE NOT FUCKING ANGLE BRACKETS!!
Copy code
provider = "keycloak-oidc"
oidc_issuer_url = "<http://keycloak:8080/realms/wizard>"
client_id = "wizard-oauth"
client_secret = "olcWi1lGcyrU82MtdvNgK8Nm7x4C8ieA"
email_domains = "*"
insecure_oidc_allow_unverified_email = true
http_address = "0.0.0.0:4180"
redirect_url = "<http://nginx:80/oauth2/callback>"
pass_access_token = true
skip_provider_button = true
reverse_proxy = true

cookie_secret = "bHEyY2E0dEZHSTlnY2hJa0FRWnRERURxaDg4Qm5qY3k="
cookie_secure = false           # IMPORTANT for HTTP; otherwise the cookie won't be sent
cookie_samesite = "lax"

set_xauthrequest = true
pass_authorization_header = true

# IMPORTANT: accept & validate incoming JWT bearer tokens for the API
skip_jwt_bearer_tokens = true
extra_jwt_issuers = [ "<http://keycloak:8080/realms/wizard=wizard-api>" ]

# Optional (handy for CORS preflight to the API)
skip_auth_preflight = true
m
✅ thought for 111.4 seconds
Totally understand the frustration — let’s get this over the line with a quick, surgical debug to prove what oauth2-proxy is actually reading and then force the right settings if needed. What the logs tell us - oauth2-proxy prints: Skipping JWT tokens from extra JWT issuer: "http://keycloak:8080/realms/wizard=wizard-api" - That means the running process sees angle brackets (or some unexpected characters) around the issuer string and is not applying it, so it falls back to validating only against the UI client_id (wizard-oauth), causing the 401. Let’s verify what the container actually sees and override if needed. 1) Verify the config inside the container Run these:
Copy code
docker compose exec oauth2-proxy sh -lc '
  echo "----- /etc/oauth2-proxy.cfg -----";
  nl -ba /etc/oauth2-proxy.cfg | sed -n "1,200p";
  echo "----- ENV -----";
  env | grep -i ^OAUTH2_PROXY || true
'
- Confirm the 
extra_jwt_issuers
line has NO angle brackets and is exactly:
Copy code
extra_jwt_issuers = [ "<http://keycloak:8080/realms/wizard=wizard-api>" ]
- If the file looks correct, check there are no env vars overriding it (printed above). 2) Ensure NGINX forwards Authorization to the auth subrequest Your /oauth2/auth block must include this line:
Copy code
proxy_set_header Authorization $http_authorization;
Without it, oauth2-proxy never sees the bearer token. 3) Force the settings via env vars (to bypass any config parsing weirdness) Add to your oauth2-proxy service in compose:
Copy code
environment:
  - OAUTH2_PROXY_SKIP_JWT_BEARER_TOKENS=true
  - OAUTH2_PROXY_EXTRA_JWT_ISSUERS=<http://keycloak:8080/realms/wizard=wizard-api>
Then restart oauth2-proxy and confirm in startup logs that the issuer is logged WITHOUT angle brackets: - Skipping JWT tokens from extra JWT issuer: http://keycloak:8080/realms/wizard=wizard-api 4) Test directly against oauth2-proxy, then through NGINX - Direct (should be 200 with a valid token):
Copy code
curl -i -H "Authorization: Bearer $TOKEN" <http://oauth2-proxy:4180/oauth2/auth>
- Through NGINX to your API (should be 200 from API):
Copy code
curl -i -H "Authorization: Bearer $TOKEN" <http://nginx/wizardous/health>
- No token should be 401 JSON (no redirect). If it still 401s after the env override: - Please paste the first 10 lines of the oauth2-proxy startup logs again - And the response from the direct check:
Copy code
curl -sv -H "Authorization: Bearer $TOKEN" <http://oauth2-proxy:4180/oauth2/auth> 2>&1 | sed -n '1,/^$/p'
Reference (for exact options and env var names): - oauth2-proxy: configuration overview
m
@Marvin
Copy code
[2025/08/31 01:23:27] [provider.go:55] Performing OIDC Discovery...

[2025/08/31 01:23:28] [provider.go:55] Performing OIDC Discovery...

[2025/08/31 01:23:28] [providers.go:145] Warning: Your provider supports PKCE methods ["plain" "S256"], but you have not enabled one with --code-challenge-method

[2025/08/31 01:23:28] [oauthproxy.go:156] Skipping JWT tokens from configured OIDC issuer: "<http://keycloak:8080/realms/wizard>"

[2025/08/31 01:23:28] [oauthproxy.go:158] Skipping JWT tokens from extra JWT issuer: "<http://keycloak:8080/realms/wizard=wizard-api>"

[2025/08/31 01:23:28] [oauthproxy.go:166] OAuthProxy configured for Keycloak OIDC Client ID: wizard-oauth

[2025/08/31 01:23:28] [oauthproxy.go:172] Cookie settings: name:_oauth2_proxy secure(https):false httponly:true expiry:168h0m0s domains: path:/ samesite:lax refresh:disabled

172.20.0.6 - c1a883e3-0564-429f-81d2-a9d0e96fa97d - - [2025/08/31 01:23:44] oauth2-proxy:4180 GET - "/oauth2/auth" HTTP/1.1 "curl/8.14.1" 401 13 0.007

[2025/08/31 01:23:44] [jwt_session.go:51] Error retrieving session from token in Authorization header: [unable to verify bearer token, could not create session from token: audience from claim aud with value [wizard-api account] does not match with any of allowed audiences map[wizard-oauth:{}], email in id_token (bbbf556f-01c1-4f63-bf4d-43cfcd60a1b6) isn't verified]

172.20.0.6 - 0610ed7a-8c30-4c80-9f1e-aca60fe828bf - - [2025/08/31 01:24:16] oauth2-proxy:4180 GET - "/oauth2/auth" HTTP/1.1 "curl/8.14.1" 401 13 0.000

[2025/08/31 01:24:16] [jwt_session.go:51] Error retrieving session from token in Authorization header: [unable to verify bearer token, could not create session from token: audience from claim aud with value [wizard-api account] does not match with any of allowed audiences map[wizard-oauth:{}], email in id_token (bbbf556f-01c1-4f63-bf4d-43cfcd60a1b6) isn't verified]
m
✅ thought for 122.5 seconds
Thanks for sticking with this — let’s force the correct behavior and remove any ambiguity. What your logs prove - oauth2-proxy is still parsing the issuer/audience with angle brackets: Skipping JWT tokens from extra JWT issuer: "http://keycloak:8080/realms/wizard=wizard-api" - When that happens, oauth2-proxy ignores your extra_jwt_issuers and validates only against the UI client_id (wizard-oauth), hence: audience … [wizard-api account] does not match allowed audiences map[wizard-oauth:{}] Let’s remove any config-file parsing surprises and hard-set the values via environment variables (these override the file). 1) Force the settings via env vars in docker compose Replace or add this environment block to your oauth2-proxy service and restart it:
Copy code
environment:
  - OAUTH2_PROXY_PROVIDER=keycloak-oidc
  - OAUTH2_PROXY_OIDC_ISSUER_URL=<http://keycloak:8080/realms/wizard>
  - OAUTH2_PROXY_CLIENT_ID=wizard-oauth
  - OAUTH2_PROXY_CLIENT_SECRET=REDACTED
  - OAUTH2_PROXY_REDIRECT_URL=<http://nginx/oauth2/callback>
  - OAUTH2_PROXY_REVERSE_PROXY=true

  # Session/cookie for UI
  - OAUTH2_PROXY_COOKIE_SECRET=REDACTED
  - OAUTH2_PROXY_COOKIE_SECURE=false
  - OAUTH2_PROXY_COOKIE_SAMESITE=lax
  - OAUTH2_PROXY_EMAIL_DOMAINS=*

  # For nginx auth_request integration
  - OAUTH2_PROXY_SET_XAUTHREQUEST=true
  - OAUTH2_PROXY_PASS_AUTHORIZATION_HEADER=true
  - OAUTH2_PROXY_PASS_ACCESS_TOKEN=true

  # CRITICAL: accept and validate Bearer JWTs for API calls
  - OAUTH2_PROXY_SKIP_JWT_BEARER_TOKENS=true
  - OAUTH2_PROXY_EXTRA_JWT_ISSUERS=<http://keycloak:8080/realms/wizard=wizard-api>

  # Optional: avoid blocking OPTIONS preflights
  - OAUTH2_PROXY_SKIP_AUTH_PREFLIGHT=true
Tip: You can temporarily remove the config file mount to ensure only env vars are used: - Remove the volume line that mounts oauth2-proxy.cfg - Or point it to an empty file This lets us confirm the env overrides work cleanly. 2) Ensure NGINX forwards the Authorization header to the auth subrequest Without this, oauth2-proxy won’t see the Bearer token.
Copy code
location = /oauth2/auth {
  proxy_pass <http://oauth2_proxy>;
  proxy_http_version 1.1;

  proxy_set_header Host               $host;
  proxy_set_header X-Real-IP          $remote_addr;
  proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto  $scheme;
  proxy_set_header X-Forwarded-Host   $host;
  proxy_set_header X-Forwarded-Uri    $request_uri;

  proxy_set_header Authorization      $http_authorization;

  proxy_set_header Content-Length     "";
  proxy_pass_request_body off;
}
And keep /wizardous/ in “no redirect” mode: - No error_page 401 = /oauth2/sign_in - On 401, return JSON (as we outlined earlier) - Forward Authorization through to your API if you want it to see the token 3) Restart and validate - Check startup logs now show the raw values (no <>): Skipping JWT tokens from extra JWT issuer: http://keycloak:8080/realms/wizard=wizard-api - Test directly against oauth2-proxy:
Copy code
curl -i -H "Authorization: Bearer $TOKEN" <http://oauth2-proxy:4180/oauth2/auth>
Expect 200 for a token with iss=http://keycloak:8080/realms/wizard and aud including wizard-api. - Then through NGINX to your API:
Copy code
curl -i -H "Authorization: Bearer $TOKEN" <http://nginx/wizardous/health>
Expect 200. No/invalid token should yield 401 JSON from NGINX. If it still fails: - Run inside the oauth2-proxy container and print effective env and config:
Copy code
docker compose exec oauth2-proxy sh -lc '
    echo "----- ENV (sanitized) -----"; env | sort | grep -E "^OAUTH2_PROXY|^OIDC|^PROVIDER" || true;
    echo "----- /etc/oauth2-proxy.cfg (if present) -----"; [ -f /etc/oauth2-proxy.cfg ] && sed -n "1,200p" /etc/oauth2-proxy.cfg || echo "(no file)";
  '
- Paste the first 10 lines of startup logs again; they should reflect the issuer without angle brackets.
Once this is green, we can circle back to any remaining polish (CORS for /wizardous/, session store in Redis to shrink cookies, etc.).
Open in Slack
PreviousNext
Powered by