fix: handle failed connection attempts once by NorthArea · Pull Request #139 · tarantool/tarantool-java-sdk

NorthArea · 2026-10-27T12:04:02Z

When a server is unreachable, silent or closes the connection before the greeting, the client's reconnect loop grows instead of making one attempt per period. This PR fixes the causes.

fix(core): report greeting timeout as a timeout — a silent server was reported both as TimeoutException and as CLOSE_BY_REMOTE, and the pool scheduled a reconnect for each. Now only TimeoutException, with the address and the timeout. The greeting timer of a previous attempt no longer closes the next one.
fix(core): skip close event before the greeting — the same double report when a server or proxy accepts and immediately closes. Close events are fired only after the greeting; the order from f7f34cd (state CLOSED first, then the event) is kept.
fix(core): register box.shutdown callback once — with graceful shutdown, every attempt added one more callback.
fix(pool): stop connect timer on failed attempt — pool.connect.time was stopped only on success.
fix(pool): pass reconnectAfter to pool entries — withReconnectAfter() and Spring Data reconnect-after had no effect; connections kept the 1000 ms default.

Each fix has a unit test that fails without it.

Если сервер недоступен, молчит или закрывает соединение до приветствия, цикл переподключения клиента разрастается вместо одной попытки за период. PR устраняет причины.

fix(core): report greeting timeout as a timeout — молчащий сервер сообщался и как TimeoutException, и как CLOSE_BY_REMOTE, и пул планировал переподключение на каждое. Теперь только TimeoutException с адресом и таймаутом. Таймер приветствия прошлой попытки больше не закрывает следующую.
fix(core): skip close event before the greeting — тот же двойной отчёт, когда сервер или прокси принимает соединение и сразу закрывает. События закрытия шлются только после приветствия; порядок из f7f34cd (сначала CLOSED, потом событие) сохранён.
fix(core): register box.shutdown callback once — при graceful shutdown каждая попытка добавляла ещё один колбэк.
fix(pool): stop connect timer on failed attempt — pool.connect.time останавливался только при успехе.
fix(pool): pass reconnectAfter to pool entries — withReconnectAfter() и reconnect-after в Spring Data не действовали, соединения оставались с умолчанием 1000 мс.

У каждого исправления есть модульный тест, который без него падает.

I haven't forgotten about:

Replaces #138

The previous probe also required no `unreachable` replica and zero unknown buckets. In CI (tests-crud-integration 3.5.0, job 122708866002) it returned false for the whole 120 s window in RepositoryViaJavaConfigTest while bootstrap had succeeded, so the container failed to start. Requiring only an `available` master is not enough either. On a stand the router state stayed `storage-002 master=unreachable replica=available` for 120 s: the router connected to storage-002-a before its storage was configured, the vshard handshake failed with "storage is not configured", and vshard closed the connection (vshard/replicaset.lua conn_vconnect_check_or_close). Only a request to that master reopens it, and discovery and failover talk to the replica. The first asynchronous crud map call reopens it but does not wait for the new handshake, which is the VHANDSHAKE_NOT_COMPLETE seen in #139. Make a synchronous `callrw('vshard.storage.buckets_count')` to the master of every replicaset: it reopens a closed connection and waits for the handshake. Require all calls to succeed and every master to be `available`, and log call errors, replica and bucket states.

The readiness probe added in 1.7.0 iterated `rs.replicas`, a field that `vshard.router.info()` does not return (vshard 0.1.25 reports `master` and `replica` per replicaset). The loop never ran, so the container was reported ready while a replicaset master was still in the vshard handshake, and the first `crud.truncate` in spring-data integration tests failed with VHANDSHAKE_NOT_COMPLETE (CI of #139). Checking the master status alone is not enough. On a stand the router state stayed `storage-002 master=unreachable replica=available` for 120 s: the router connected to storage-002-a before its storage was configured, the vshard handshake failed with "storage is not configured", and vshard closed the connection (vshard/replicaset.lua conn_vconnect_check_or_close). Only a request to that master reopens it; discovery and failover talk to the replica. The first asynchronous crud map call reopens it but does not wait for the new handshake. Requiring zero unknown buckets does not work either: the router may not know part of the buckets for a while after the masters are ready. Make a synchronous `callrw('vshard.storage.buckets_count')` to the master of every replicaset: it reopens a closed connection and waits for the handshake. Require all calls to succeed and every master to be `available`, and log call errors, replica and bucket states.

When a server accepts the TCP connection but does not send the greeting, the greeting timer closed the channel without setting a close reason. The close handler then treated it as a close by the server: it fired CLOSE_BY_REMOTE with "Connection closed by server" next to the TimeoutException of the connect future. The pool listens to both, so every failed attempt was handled twice and the reconnect chains multiplied: in one run of the new pool test, 12 attempts to a silent server scheduled 66 reconnects instead of one per attempt. Fail the connect promise with TimeoutException first and close the channel only if that succeeded, so a connection whose greeting has just arrived is never closed by the timer. Close it with a dedicated reason and do not fire a close event for it. Name the address and the timeout in the message. The timer also stayed armed when the server closed the connection before the greeting, and then fired during the next attempt and closed its channel: in two runs of the new test the second attempt was closed 220 and 326 ms after it started, with a 600 ms timeout. Let the timer close the channel only of the attempt it was scheduled for.

When a server, or a proxy in front of it, accepted the TCP connection and closed it before the greeting, the close was reported twice: the connect future failed with ConnectionClosedException, and the close handler also fired CLOSE_BY_REMOTE. The pool listens to both and scheduled a reconnect for each, so the reconnect chains multiplied: in one run of the new pool test, 50 attempts in 3 s scheduled 1325 reconnects. A connection that has not received the greeting has no requests in flight and no registered watchers, and its failure already reaches the caller through the connect future. Fire close events only for a connection that has received the greeting.

With graceful shutdown enabled, IProtoClientImpl.connect() called watch("box.shutdown", this::shutdownEventCallback) on every attempt. watch() appends the callback to the existing watcher and nothing removes it, so a client that kept reconnecting to an unavailable server accumulated one callback per attempt, and the event after a successful connect ran the callback once per previous attempt. The watcher itself survives reconnects and is re-subscribed by updateWatchers(), so add the callback only on the first connect.

The pool.connect.time sample was stopped only when the connect future succeeded, so every failed attempt left an active sample behind: in the new test, 12 attempts to a silent server left 12 active samples. Stop the sample when the connect future completes either way.

TarantoolClientImpl calls pool.setGroups() before pool.setReconnectAfter(), so the connections were created with the pool default of 1000 ms. setReconnectAfter() changed only the pool field, and PoolEntry kept its final copy, so withReconnectAfter() and the reconnect-after property of Spring Data had no effect on these connections. Pass the new value to existing connections, as setConnectTimeout() already does.

bitgorbovsky · 2026-10-29T17:22:30Z


-  private void scheduleGreetingTimeout(CompletableFuture<Greeting> promise, long timeoutMs) {
+  private void scheduleGreetingTimeout(
+      CompletableFuture<Greeting> promise, InetSocketAddress address, long timeoutMs) {


Мне кажется, что добавлять аргумент address только ради того, чтобы в сообщении исключения был адрес - лишнее, поскольку клиентский код, который вызывает connect(), он уже знает собственно адрес и может правильно растолковать, о чём же была ошибка ожидания.

bitgorbovsky · 2026-10-29T17:37:24Z

+
+  @Test
+  public void test_greetingTimeout_shouldNotBeReportedAsCloseByServer() throws Exception {
+    List<Throwable> closedByRemote = new CopyOnWriteArrayList<>();


Просто замечание: почему-то нейронки очень любят CopyOnWriteArraList, хотя можно просто вписать LinkedList :)

bitgorbovsky · 2026-10-29T18:01:01Z

Посмотри пожалуйста вот туда: https://github.com/tarantool/tarantool-java-sdk/blob/master/tarantool-core/src/test/java/io/tarantool/core/integration/ConnectionToTarantoolTest.java

Там уже есть тесты на greeting, в том числе и на timeout.
Стоит доработать именно эти тесты, чтобы не было разброда и шатания

NorthArea mentioned this pull request Oct 27, 2026

fix: handle failed connection attempts once #138

Closed

6 tasks done

nickkkccc requested review from bitgorbovsky and nickkkccc October 27, 2026 12:40

NorthArea force-pushed the northarea/reconnect-loop-fixes branch from 7325a00 to 3273756 Compare October 27, 2026 14:05

NorthArea mentioned this pull request Oct 28, 2026

fix: stabilize flaky integration tests #140

Merged

4 of 6 tasks

NorthArea force-pushed the northarea/reconnect-loop-fixes branch 2 times, most recently from 049a7f1 to 94dfd02 Compare October 29, 2026 13:48

Aleksandr Iuskasov added 5 commits October 29, 2026 19:42

NorthArea force-pushed the northarea/reconnect-loop-fixes branch from e6781c9 to f0d30b1 Compare October 29, 2026 15:15

bitgorbovsky requested changes Oct 29, 2026

View reviewed changes

Navigation Menu

Repository navigation

fix: handle failed connection attempts once - #139

fix: handle failed connection attempts once#139
NorthArea wants to merge 5 commits into
masterfrom
northarea/reconnect-loop-fixes

NorthArea commented Oct 27, 2026 •

edited

Loading

Uh oh!

bitgorbovsky Oct 29, 2026

Uh oh!

bitgorbovsky Oct 29, 2026

Uh oh!

bitgorbovsky Oct 29, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

NorthArea commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bitgorbovsky Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

bitgorbovsky Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

bitgorbovsky Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

NorthArea commented Oct 27, 2026 •

edited

Loading