problem on adding new node to kubernetes

[ugenk]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Kubernetes

CLOUDSTACK VERSION

4.19.0.1

CONFIGURATION

Network offering: DefaultNetworkOfferingforKubernetesService

OS / ENVIRONMENT

N/A

SUMMARY

While resizing k8s cluster, we're getting error related to adding firewall rules.

STEPS TO REPRODUCE

We're reproducing that constantly on our installed cluster. Please, share which information do you need to help you to reproduce that.

Here is our firewall and portforwarding rules for that SNAT IP:

mysql> select * from firewall_rules where ip_address_id = 4433;
+-------+--------------------------------------+---------------+------------+----------+--------+----------+----------------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
| id    | uuid                                 | ip_address_id | start_port | end_port | state  | protocol | purpose        | account_id | domain_id | network_id | xid                                  | created             | icmp_code | icmp_type | related | type | vpc_id | traffic_type | display |
+-------+--------------------------------------+---------------+------------+----------+--------+----------+----------------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
|   157 | 160544dc-303a-4449-9ee5-9aa7c74ecd8e |          4433 |       6443 |     6443 | Active | tcp      | Firewall       |         23 |         1 |        267 | 592c3df8-3804-4ccb-8030-f1a16c77e431 | 2023-05-27 08:18:26 |      NULL |      NULL |    NULL | User |   NULL | Ingress      |       1 |
|   159 | 53a32814-ab9a-413d-9fba-834c60cbe41c |          4433 |       6443 |     6443 | Active | tcp      | LoadBalancing  |         23 |         1 |        267 | 8bfbb10f-5577-4a22-b20e-19f7a6c155ff | 2023-05-27 08:18:29 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
|   169 | 11c33409-788b-4639-9e77-39f124a97cd8 |          4433 |       3222 |     3222 | Active | tcp      | PortForwarding |         23 |         1 |        267 | b3c13c77-0d29-4fe1-9d6d-6e023acab91c | 2023-05-29 20:28:54 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
|   171 | 97f9a76f-36f1-4f2b-80da-6fedf100edcb |          4433 |       3222 |     3222 | Active | tcp      | Firewall       |         23 |         1 |        267 | aac0f2b6-488b-4e6b-aff7-47e26dc588f8 | 2023-05-29 20:32:47 |      NULL |      NULL |    NULL | User |   NULL | Ingress      |       1 |
|   202 | ede86b93-2fab-4662-a75f-a7eaee50c736 |          4433 |       2222 |     2222 | Active | tcp      | PortForwarding |         23 |         1 |        267 | e1645b58-8430-42e4-9108-2f474e71dc34 | 2023-05-31 13:12:51 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
|   203 | 1ae4c78a-5736-4a14-b22d-5dc4876f3ebc |          4433 |       2223 |     2223 | Active | tcp      | PortForwarding |         23 |         1 |        267 | 35d2cbb1-9644-4a2c-ab59-e9de8cf32219 | 2023-05-31 13:12:52 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
|   204 | 2ae6750d-5d51-4812-b60f-abdba5c8cc37 |          4433 |       2224 |     2224 | Active | tcp      | PortForwarding |         23 |         1 |        267 | d20b18a2-9287-415a-bcd2-69cf4c08414b | 2023-05-31 13:12:54 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
|   205 | 315e1cb9-f0d2-438c-92b2-babbb346b8b9 |          4433 |       2225 |     2225 | Active | tcp      | PortForwarding |         23 |         1 |        267 | a971ffbe-cf56-4191-8589-8f660116124c | 2023-05-31 13:12:55 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
|   206 | 3cc5707a-a5f0-4eaa-98db-31190e4f777a |          4433 |       2226 |     2226 | Active | tcp      | PortForwarding |         23 |         1 |        267 | de06f445-9db3-4652-8ef4-6dc82e2f36ef | 2023-05-31 13:12:56 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
|   207 | a568da68-a0a8-4eae-9e12-8ed0ebd55a8a |          4433 |       2227 |     2227 | Active | tcp      | PortForwarding |         23 |         1 |        267 | 4e7bebda-591c-4051-bab3-a829ee3e888c | 2023-05-31 13:12:57 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
|   210 | 878750f8-cdaa-42bf-af6a-46a5d531eab3 |          4433 |       3223 |     3223 | Active | tcp      | Firewall       |         23 |         1 |        267 | 4a497f27-af34-4ff5-9d8f-77be06cf080f | 2023-05-31 17:10:58 |      NULL |      NULL |    NULL | User |   NULL | Ingress      |       1 |
|   211 | 40785ce1-ca7d-46f3-ac87-45c52dccd078 |          4433 |       3223 |     3223 | Active | tcp      | PortForwarding |         23 |         1 |        267 | 45fff41a-8d5e-446f-9766-03ce2ffc522a | 2023-05-31 17:11:29 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
|   259 | 227e9e80-008f-4598-8474-f983d559d355 |          4433 |       3224 |     3224 | Active | tcp      | PortForwarding |         23 |         1 |        267 | 3086de53-2d84-49a5-b52d-b62a6ac23943 | 2023-06-21 13:57:34 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
|   260 | f93c69ce-598e-4d5d-b9d6-a122bcb937da |          4433 |       3224 |     3224 | Active | tcp      | Firewall       |         23 |         1 |        267 | b9d22186-6bbd-4eed-a864-97fe3e44698e | 2023-06-21 13:58:08 |      NULL |      NULL |    NULL | User |   NULL | Ingress      |       1 |
|   281 | 21962db8-97f1-4709-8085-70bfa5cfa47f |          4433 |      35432 |    35432 | Active | tcp      | PortForwarding |         23 |         1 |        267 | 5eddfc85-fec9-48b3-b4e6-8f2a44669c43 | 2023-06-23 09:32:01 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
|   282 | a2e99cd4-a3e9-493f-a461-d3827b5b8c6d |          4433 |      35432 |    35432 | Active | tcp      | Firewall       |         23 |         1 |        267 | 416c60de-3b8a-4891-8bd7-729b6ea31fb1 | 2023-06-23 09:39:29 |      NULL |      NULL |    NULL | User |   NULL | Ingress      |       1 |
|   284 | 8505e874-127b-4523-af26-0c4462608a3f |          4433 |       NULL |     NULL | Active | icmp     | Firewall       |         23 |         1 |        267 | 5f2e4b46-91f6-4c1f-a24f-5480cb63d052 | 2023-07-02 10:25:40 |        -1 |        -1 |    NULL | User |   NULL | Ingress      |       1 |
|  1470 | ae9602cb-ad43-4251-8b1d-9ca0341a6109 |          4433 |      32727 |    32727 | Active | tcp      | LoadBalancing  |         23 |         1 |        267 | 11ace4c3-7c0e-4ec7-ae22-44dd06e8ce5c | 2023-07-08 13:20:19 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
|  1471 | 6a2753e3-3f87-4183-b541-3274dac0c724 |          4433 |      32727 |    32727 | Active | tcp      | Firewall       |         23 |         1 |        267 | ed955550-d415-4f8d-8460-4983f5b8fb29 | 2023-07-08 13:20:19 |      NULL |      NULL |    NULL | User |   NULL | Ingress      |       1 |
|  3211 | 144fbdaa-b95e-4a86-ad8b-bc777e87e7c3 |          4433 |       5601 |     5601 | Active | tcp      | PortForwarding |         23 |         1 |        267 | f39579fb-8d84-448b-86e9-c77939b42e95 | 2023-07-14 12:19:22 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
|  3213 | 1bc0cef8-dace-48f8-b0ae-bda9a9451797 |          4433 |          1 |    65535 | Active | tcp      | Firewall       |         23 |         1 |        267 | c3bd9b08-9957-481e-a81a-ef470ac744f6 | 2023-07-14 12:21:51 |      NULL |      NULL |    NULL | User |   NULL | Ingress      |       1 |
|  3214 | b5b6b139-839a-47d0-80da-8f6b775855c4 |          4433 |          1 |    65535 | Active | tcp      | Firewall       |         23 |         1 |        267 | ef6eb373-b177-4f5b-bc42-3b9604629f67 | 2023-07-14 12:22:11 |      NULL |      NULL |    NULL | User |   NULL | Ingress      |       1 |
|  3223 | cee5dc3d-a53e-43a1-ab52-cfadfd223382 |          4433 |          1 |    65535 | Active | tcp      | Firewall       |         23 |         1 |        267 | 0ca97b93-9784-4c5b-b06d-22079340f8dc | 2023-07-14 13:04:01 |      NULL |      NULL |    NULL | User |   NULL | Ingress      |       1 |
| 15760 | 4fafe230-fc83-4540-ba11-271c0d717c75 |          4433 |       2228 |     2230 | Active | tcp      | Firewall       |         23 |         1 |        267 | 48f8bf55-cc62-4faa-a95d-bd244dbfcf98 | 2023-09-14 08:25:01 |      NULL |      NULL |    NULL | User |   NULL | Ingress      |       1 |
| 39225 | 9087fdc2-1f6c-4ec3-902e-a7cd1b7b8b08 |          4433 |       2228 |     2228 | Active | tcp      | PortForwarding |         23 |         1 |        267 | 69417616-fd55-40f6-bfcd-14b2a21b6137 | 2023-12-05 10:02:08 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
| 48706 | 910ced9a-965e-40cf-94d4-1cb40507e576 |          4433 |          1 |    65535 | Active | tcp      | Firewall       |         23 |         1 |        267 | 43342855-4472-4686-b44b-7709545163a2 | 2024-01-09 02:40:01 |      NULL |      NULL |    NULL | User |   NULL | Ingress      |       1 |
| 48707 | 4ed50fde-a0c9-4b4b-8bad-7d6d9eb95948 |          4433 |       2929 |     2929 | Active | tcp      | Firewall       |         23 |         1 |        267 | f33949bd-bb92-4ead-85df-1b18ae26ad0f | 2024-01-17 03:23:08 |      NULL |      NULL |    NULL | User |   NULL | Ingress      |       1 |
| 48708 | 8b351e99-4c26-402b-971c-11decac854ff |          4433 |        500 |      500 | Staged | udp      | Vpn            |         23 |         1 |        267 | bc287688-aedc-4911-a7fb-6911327855c8 | 2024-01-21 06:13:47 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
| 48709 | af332d08-8cb7-4d78-b466-fadc001cc26c |          4433 |        500 |      500 | Active | udp      | Firewall       |         23 |         1 |        267 | 27884745-0c75-455f-9a75-ce6e6d88c325 | 2024-01-21 06:13:47 |      NULL |      NULL |   48708 | User |   NULL | Ingress      |       1 |
| 48710 | 6d7c444b-1044-48b6-99b1-a7c125fa479c |          4433 |       1701 |     1701 | Staged | udp      | Vpn            |         23 |         1 |        267 | 433eaf8d-a020-4899-8aab-1c2777871e5f | 2024-01-21 06:13:47 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
| 48711 | 3fa7d224-1608-4125-83e4-7107388ad66c |          4433 |       1701 |     1701 | Active | udp      | Firewall       |         23 |         1 |        267 | 9b486a5b-a359-47dc-88cc-f760dfd44a2d | 2024-01-21 06:13:47 |      NULL |      NULL |   48710 | User |   NULL | Ingress      |       1 |
| 48712 | bd0aa221-32f0-48d5-8f7a-5bbe4e3a8dbe |          4433 |       4500 |     4500 | Staged | udp      | Vpn            |         23 |         1 |        267 | 580ceedd-d867-41d7-93f2-e50737f9ed97 | 2024-01-21 06:13:47 |      NULL |      NULL |    NULL | User |   NULL | NULL         |       1 |
| 48713 | b7c9636c-5421-45fc-b779-e600e1c24db8 |          4433 |       4500 |     4500 | Active | udp      | Firewall       |         23 |         1 |        267 | bfe7a1bc-c862-41a1-bdd6-e9274716310b | 2024-01-21 06:13:47 |      NULL |      NULL |   48712 | User |   NULL | Ingress      |       1 |
+-------+--------------------------------------+---------------+------------+----------+--------+----------+----------------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
33 rows in set (0.00 sec)

EXPECTED RESULTS

Fix the problem

ACTUAL RESULTS

2024-04-18 22:07:11,414 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-1:ctx-d42166a9 job-4626/job-4627) (logid:e142ff5b) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 392, job origin: 4626
2024-04-18 22:07:11,415 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-d42166a9 job-4626/job-4627) (logid:e142ff5b) Done executing com.cloud.vm.VmWorkStart for job-4627
2024-04-18 22:07:11,417 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-1:ctx-d42166a9 job-4626/job-4627) (logid:e142ff5b) Remove job-4627 from job monitoring
2024-04-18 22:07:11,421 INFO  [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-1:ctx-7a5b66c2 job-4626 ctx-9e48efab) (logid:e142ff5b) Started VM : mediatech-node-18ef29acb54 in the Kubernetes cluster : mediatech
2024-04-18 22:07:11,424 INFO  [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-1:ctx-7a5b66c2 job-4626 ctx-9e48efab) (logid:e142ff5b) Provisioned node VM : mediatech-node-18ef29acb54 in to the Kubernetes cluster : mediatech
2024-04-18 22:07:11,435 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-1:ctx-7a5b66c2 job-4626) (logid:e142ff5b) Unexpected exception while executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd
java.lang.NullPointerException
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.removeSshFirewallRule(KubernetesClusterResourceModifierActionWorker.java:520)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterIsolatedNetworkRules(KubernetesClusterScaleWorker.java:124)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterNetworkRules(KubernetesClusterScaleWorker.java:174)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleUpKubernetesClusterSize(KubernetesClusterScaleWorker.java:376)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterSize(KubernetesClusterScaleWorker.java:412)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleCluster(KubernetesClusterScaleWorker.java:465)
	at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.scaleKubernetesCluster(KubernetesClusterManagerImpl.java:1548)
	at org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd.execute(ScaleKubernetesClusterCmd.java:156)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
2024-04-18 22:07:11,448 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-7a5b66c2 job-4626) (logid:e142ff5b) Complete async job-4626, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530"}

[ugenk]

The exception originates from here:
https://github.com/apache/cloudstack/blob/4.19.0.1/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterResourceModifierActionWorker.java#L520
Has it been addressed in #8107

[DaanHoogland]

@ugenk this code has been merged and should be in 4.19.1 as well, Have you tested with 4.18.2 or a newer version of 4.19?
The description indicates that it is fixed.

[ugenk]

we have 4.19.0.1, so we have to wait 4.19.1 release i'm afraid

[weizhouapache]

we have 4.19.0.1, so we have to wait 4.19.1 release i'm afraid

you can cherry-pick the commit to 4.19.0.1, build the packages and deploy it. @ugenk
cloudstack is a open-source project, you can build whenever needed.

[yadvr]

Fixed in #8107