Problem Description:
Hi Team,
Project upgraded consul from 1.17.3 → 1.22.3, after upgrade existing ACL tokens in running pods became invalid(ACL not found). Their microservice pods were not restarted, so they kept using the old in-memory token.
consul-config-provider refreshes the consul token only by TTL(about 5 minutes before expiration). It does not detect that consul no longer knows the token after an infra upgrade/re-bootstrap.
Spring Cloud Consul ConfigWatch keeps calling KV watch with that dead token. On 403 consul returns immediately(no blocking wait-time) and the next attempt runs after watch.delay(default 1s). With ~200 services this overloads consul and causes 429 Too Many Requests.
A pod restart fixes it because startup does /v1/acl/login again and gets a new token. There is currently no recovery without restart.
Quarkus is not affected (error sleep is already 10s). Spring services can reduce the flood with spring.cloud.consul.config.watch.delay, but that does not refresh the token.
Root cause
TokenUpdater schedules re-login only from ExpirationTime. After Consul re-bootstrap the token is invalid immediately, while TTL is still in the future, so no re-login happens.
ConfigWatch reads ConsulConfigProperties.aclToken on each poll. ConfigBasedTokenStorage.update() already writes a new token there. The missing piece is detecting an invalid token and calling /v1/acl/login at runtime.
Expectations:
Can something be done to refresh this token without restart of microservice?
or Can we add logic like it will periodically validate the current token via GET /v1/acl/token/self
and On 403/ACL not found(or any validation failure), request a new token via /v1/acl/login and update aclToken. Keep existing TTL refresh(~5 minutes before expiry).
Problem Description:
Hi Team,
Project upgraded consul from 1.17.3 → 1.22.3, after upgrade existing ACL tokens in running pods became invalid(ACL not found). Their microservice pods were not restarted, so they kept using the old in-memory token.
consul-config-provider refreshes the consul token only by TTL(about 5 minutes before expiration). It does not detect that consul no longer knows the token after an infra upgrade/re-bootstrap.
Spring Cloud Consul ConfigWatch keeps calling KV watch with that dead token. On 403 consul returns immediately(no blocking wait-time) and the next attempt runs after watch.delay(default 1s). With ~200 services this overloads consul and causes 429 Too Many Requests.
A pod restart fixes it because startup does /v1/acl/login again and gets a new token. There is currently no recovery without restart.
Quarkus is not affected (error sleep is already 10s). Spring services can reduce the flood with spring.cloud.consul.config.watch.delay, but that does not refresh the token.
Root cause
TokenUpdater schedules re-login only from ExpirationTime. After Consul re-bootstrap the token is invalid immediately, while TTL is still in the future, so no re-login happens.
ConfigWatch reads ConsulConfigProperties.aclToken on each poll. ConfigBasedTokenStorage.update() already writes a new token there. The missing piece is detecting an invalid token and calling /v1/acl/login at runtime.
Expectations:
Can something be done to refresh this token without restart of microservice?
or Can we add logic like it will periodically validate the current token via GET /v1/acl/token/self
and On 403/ACL not found(or any validation failure), request a new token via /v1/acl/login and update aclToken. Keep existing TTL refresh(~5 minutes before expiry).