Keycloak 25.0.0 Helm Chart deployment -> CrashLoop

[zeppelinux]

Upgraded keycloak from 22.0..4 to 25.0.0 using new keycloakx chart as a drop in replacement - same helm cmd line, same values.yaml. The service starts and fails/CrashLoops.
Initial error:
Startup probe failed: Get "http://10.42.2.163:9000/auth/health": dial tcp 10.42.2.163:9000: connect: connection refused

I disabled health check by adding health.enabled=false to the values.yaml, but it still crashes. Nothing in the service logs besides this:

`Appending additional Java properties to JAVA_OPTS
Changes detected in configuration. Updating the server image.
Updating the configuration and installing your custom providers, if any. Please wait.
2024-06-29 00:19:35,792 INFO [io.qua.dep.QuarkusAugmentor] (main) Quarkus augmentation completed in 16683ms
Server configuration updated and persisted. Run the following command to review the configuration:
kc.sh show-config
Next time you run the server, just run:
kc.sh --verbose start --http-enabled=true --http-port=8080 --hostname-strict=false --hostname-strict-https=false --spi-events-listener-jboss-logging-success-level=info --spi-events-listener-jboss-logging-error-level=warn --optimized
Disabled option: '--hostname-strict-https'. Available only when hostname:v1 feature is enabled
Possible solutions: --hostname, --hostname-admin, --hostname-backchannel-dynamic, --hostname-strict, --hostname-debug
Try 'kc.sh start --help' for more information on the available options.
Specify '--help-all' to obtain information on all options and their availability

DB checker:

Waiting for Database to become ready... .Database OK ✓

CrashLoopBackOff: back-off 5m0s restarting failed container=keycloak

[linonetwo]

See mine, mine no crash, but web console not loaded #777

[zeppelinux]

See mine, mine no crash, but web console not loaded #777

I don't have any error anywhere i can see though, so it is different.

[chadm-airia]

@zeppelinux - did you find a resolution to this? I'm seeing the same issue. I had a good install with 22.0.4, but I bumped the image to 24.0.5. Same problem, though.

[chadm-airia]

I had to replace my startupProbe, readinessProbe, and livenessProbe in my release. Essentially, all of the probes were made available on http instead of http-internal and nothing I did would change it.

After I added the probes to my values.yaml with the port set to http instead of http-internal, it worked fine.

[AMontagu]

The issue come from you are using the --hostname-strict-https options that has been removed.

Please see: #779

[ulevitsky]

I was facing the same problem with all probes failing. The root cause was that management interface only listens on HTTPS but not HTTP. Apparently, unlike the main KC server, management interface server, which is now a separate server, can only listen on one but not both.

I've tried a lot of things to get it to listen on HTTP instead of HTTPS, but couldn't figure out how. Stuck with this for now:

2024-07-11 01:11:27,725 INFO [io.quarkus] (main) Keycloak 25.0.1 on JVM (powered by Quarkus 3.8.5) started in 14.420s. Listening on: http://0.0.0.0:8080/ and https://0.0.0.0:8443./ Management interface listening on https://0.0.0.0:9000./

For reference, this is my startup command:

  command:
    - "/opt/keycloak/bin/kc.sh"
    - "start"
    - "--hostname=keycloak"
    - "--http-enabled=true"
    - "--http-port=8080"
    - "--hostname-strict=false"
    - "--hostname-backchannel-dynamic=false"

For want of a better solution, I got probes to work for now by redeclaring the default probes from the template in my values.yaml but with scheme: HTTPS added, like so:

  livenessProbe: |
    httpGet:
      scheme: HTTPS
      path: '{{ tpl .Values.http.relativePath $ | trimSuffix "/" }}/health/live'
      port: http-internal
    initialDelaySeconds: 0
    timeoutSeconds: 5

  readinessProbe: |
    httpGet:
      scheme: HTTPS
      path: '{{ tpl .Values.http.relativePath $ | trimSuffix "/" }}/health/ready'
      port: http-internal
    initialDelaySeconds: 10
    timeoutSeconds: 1

  startupProbe: |
    httpGet:
      scheme: HTTPS
      path: '{{ tpl .Values.http.relativePath $ | trimSuffix "/" }}/health'
      port: http-internal
    initialDelaySeconds: 20
    timeoutSeconds: 1
    failureThreshold: 60
    periodSeconds: 5

Fortunately, K8s doesn't care about TLS certificate validity when it comes to HTTP probes.

I hope it helps someone.

[beezerk23]

Hey, im also running into a crashloop. For whatever reason, all my deployment is doing is this:

Appending additional Java properties to JAVA_OPTS
Updating the configuration and installing your custom providers, if any. Please wait.
2024-07-11 12:38:47,833 INFO  [io.qua.dep.QuarkusAugmentor] (main) Quarkus augmentation completed in 110325ms
2024-07-11 12:39:44,173 INFO  [org.infinispan.CONTAINER] (Thread-5) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2024-07-11 12:39:48,152 INFO  [org.infinispan.CLUSTER] (Thread-5) ISPN000078: Starting JGroups channel `ISPN` with stack `kubernetes`
2024-07-11 12:39:48,162 INFO  [org.jgroups.JChannel] (Thread-5) local_addr: 80abec56-ff79-4a91-b44d-2799b0a24108, name: keycloak-keycloakx-0-61709
2024-07-11 12:39:48,265 INFO  [org.jgroups.protocols.FD_SOCK2] (Thread-5) server listening on *.57800
2024-07-11 12:39:50,311 INFO  [org.jgroups.protocols.pbcast.GMS] (Thread-5) keycloak-keycloakx-0-61709: no members discovered after 2008 ms: creating cluster as coordinator
2024-07-11 12:39:50,690 INFO  [org.infinispan.CLUSTER] (Thread-5) ISPN000094: Received new cluster view for channel ISPN: [keycloak-keycloakx-0-61709|0] (1) [keycloak-keycloakx-0-61709]
2024-07-11 12:39:54,320 INFO  [org.infinispan.CLUSTER] (Thread-5) ISPN000079: Channel `ISPN` local address is `keycloak-keycloakx-0-61709`, physical addresses are `[10.8.2.106:7800]`
2024-07-11 12:39:57,583 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2024-07-11 12:39:59,916 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: keycloak-keycloakx-0-61709, Site name: null
2024-07-11 12:40:16,896 INFO  [io.quarkus] (main) Keycloak 25.0.1 on JVM (powered by Quarkus 3.8.5) started in 88.214s. Listening on: http://0.0.0.0:8080. Management interface listening on http://0.0.0.0:9000.
2024-07-11 12:40:16,906 INFO  [io.quarkus] (main) Profile dev activated.
2024-07-11 12:40:16,912 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-mysql, keycloak, logging-gelf, micrometer, narayana-jta, reactive-routes, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, smallrye-health, vertx]
2024-07-11 12:40:16,989 WARN  [org.keycloak.quarkus.runtime.KeycloakMain] (main) Running the server in development mode. DO NOT use this configuration in production.
2024-07-11 12:41:45,889 INFO  [org.infinispan.CLUSTER] (Thread-12) ISPN000080: Disconnecting JGroups channel `ISPN`
2024-07-11 12:41:46,031 INFO  [com.arjuna.ats.jbossatx] (main) ARJUNA032014: Stopping transaction recovery manager
2024-07-11 12:41:46,172 INFO  [io.quarkus] (main) Keycloak stopped in 0.557s

As you can see after some time keycloak just stops. I use 2.4.3 of the helm chart and 25.0.1 as the keycloak version. In my kubernetes logs i get this:

Startup probe failed: HTTP probe failed with statuscode: 404
Startup probe failed: Get "http://10.8.2.106:8080/health": dial tcp 10.8.2.106:8080: connect: connection refused

This is my start command configured in the helm values:

command:
  - "/opt/keycloak/bin/kc.sh"
  - "start-dev"
  - "--verbose"
  - "--hostname=https://MY_DOMAIN"
  - "--http-port=8080"
  - "--hostname-strict=false"

Any help would be highly appreciated.