HIVE-29709: Add health check to docker

[ayushtkn]

What changes were proposed in this pull request?

Add health check to Hive docker startup

Why are the changes needed?

Better user experiance

Does this PR introduce any user-facing change?

Yes, post hive cluster startup, you need not to wait randomly

How was this patch tested?

Manually

Before

[+] Building 0.0s (0/0)
[+] Running 7/7
 ✔ Network hive             Created                                                                                                                                                     0.0s
 ✔ Volume "hive_hive-db"    Created                                                                                                                                                     0.0s
 ✔ Volume "hive_warehouse"  Created                                                                                                                                                     0.0s
 ✔ Volume "hive_scratch"    Created                                                                                                                                                     0.0s
 ✔ Container postgres       Started                                                                                                                                                     1.4s
 ✔ Container metastore      Started                                                                                                                                                     1.1s
 ✔ Container hiveserver2    Started                                                                                                                                                     1.1s
ayushsaxena@Q3NW54Y0C5 docker % docker exec -it hiveserver2 beeline -u 'jdbc:hive2://hiveserver2:10000/'
Connecting to jdbc:hive2://hiveserver2:10000/
26/07/06 14:22:15 [main]: WARN jdbc.HiveConnection: Failed to connect to hiveserver2:10000
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status. Enable verbose error messages (--verbose=true) for more information.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://hiveserver2:10000/: java.net.ConnectException: Connection refused (state=08S01,code=0)
ayushsaxena@Q3NW54Y0C5 docker % docker exec -it hiveserver2 beeline -u 'jdbc:hive2://hiveserver2:10000/'
Connecting to jdbc:hive2://hiveserver2:10000/
26/07/06 14:22:20 [main]: WARN jdbc.HiveConnection: Failed to connect to hiveserver2:10000
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status. Enable verbose error messages (--verbose=true) for more information.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://hiveserver2:10000/: java.net.ConnectException: Connection refused (state=08S01,code=0)
ayushsaxena@Q3NW54Y0C5 docker % docker exec -it hiveserver2 beeline -u 'jdbc:hive2://hiveserver2:10000/'
Connecting to jdbc:hive2://hiveserver2:10000/
26/07/06 14:22:25 [main]: WARN jdbc.HiveConnection: Failed to connect to hiveserver2:10000
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status. Enable verbose error messages (--verbose=true) for more information.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://hiveserver2:10000/: java.net.ConnectException: Connection refused (state=08S01,code=0)
ayushsaxena@Q3NW54Y0C5 docker % docker exec -it hiveserver2 beeline -u 'jdbc:hive2://hiveserver2:10000/'
Connecting to jdbc:hive2://hiveserver2:10000/
26/07/06 14:22:37 [main]: WARN jdbc.HiveConnection: Failed to connect to hiveserver2:10000
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status. Enable verbose error messages (--verbose=true) for more information.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://hiveserver2:10000/: java.net.ConnectException: Connection refused (state=08S01,code=0)

After - Non-LLAP

[+] Building 0.0s (0/0)
[+] Running 7/7
 ✔ Network hive             Created                                                                                                                                                     0.0s
 ✔ Volume "hive_scratch"    Created                                                                                                                                                     0.0s
 ✔ Volume "hive_hive-db"    Created                                                                                                                                                     0.0s
 ✔ Volume "hive_warehouse"  Created                                                                                                                                                     0.0s
 ✔ Container postgres       Healthy                                                                                                                                                    17.2s
 ✔ Container metastore      Healthy                                                                                                                                                    17.2s
 ✔ Container hiveserver2    Healthy                                                                                                                                                    27.2s
Hive cluster is ready.
ayushsaxena@Q3NW54Y0C5 docker % docker exec -it hiveserver2 beeline -u 'jdbc:hive2://hiveserver2:10000/'
Connecting to jdbc:hive2://hiveserver2:10000/
Connected to: Apache Hive (version 4.3.0-SNAPSHOT)
Driver: Hive JDBC (version 4.3.0-SNAPSHOT)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 4.3.0-SNAPSHOT by Apache Hive
0: jdbc:hive2://hiveserver2:10000/>

After - LLAP

+] Building 0.0s (0/0)
[+] Running 14/14
 ✔ Network hive                 Created                                                                                                                                                 0.0s
 ✔ Volume "zookeeper_datalog"   Created                                                                                                                                                 0.0s
 ✔ Volume "zookeeper_logs"      Created                                                                                                                                                 0.0s
 ✔ Volume "hive_hive-db"        Created                                                                                                                                                 0.0s
 ✔ Volume "hive_warehouse"      Created                                                                                                                                                 0.0s
 ✔ Volume "hive_scratch"        Created                                                                                                                                                 0.0s
 ✔ Volume "zookeeper_data"      Created                                                                                                                                                 0.0s
 ✔ Container zookeeper          Healthy                                                                                                                                                17.5s
 ✔ Container postgres           Healthy                                                                                                                                                17.5s
 ✔ Container hive-llapdaemon-2  Healthy                                                                                                                                                17.4s
 ✔ Container tezam              Healthy                                                                                                                                                17.4s
 ✔ Container hive-llapdaemon-1  Healthy                                                                                                                                                17.4s
 ✔ Container metastore          Healthy                                                                                                                                                17.4s
 ✔ Container hiveserver2        Healthy                                                                                                                                                27.4s
Hive cluster is ready.
ayushsaxena@Q3NW54Y0C5 docker % docker exec -it hiveserver2 beeline -u 'jdbc:hive2://hiveserver2:10000/'
Connecting to jdbc:hive2://hiveserver2:10000/
Connected to: Apache Hive (version 4.3.0-SNAPSHOT)
Driver: Hive JDBC (version 4.3.0-SNAPSHOT)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 4.3.0-SNAPSHOT by Apache Hive
0: jdbc:hive2://hiveserver2:10000/>

[abstractdog]

very nice improvement @ayushtkn , 2 comments only

[abstractdog]

for clarity's sake, zookeeper is supposed to return "imok", however this happens after start:

docker compose exec zookeeper sh -c 'printf ruok | nc -w 2 127.0.0.1 2181'
...
ruok is not executed because it is not in the whitelist.

maybe

environment:
      ZOO_4LW_COMMANDS_WHITELIST: ruok

I think if the zookeeper is unable to start, then a connection refused is already enough to ensure the unhealthy state, however, it would be good to somehow have "imok" in the response check

[abstractdog]

is there a chance to have healthcheck for tezams?
tez.am.rpc.port could be set to a static value, and multiple tez AM's won't collide given they are on different hosts like llap daemons

https://github.com/apache/tez/blob/4bccf3a2339c913ade0bc5447bdfc505c5536128/tez-dag/src/main/java/org/apache/tez/dag/api/client/DAGClientServer.java#L69

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud