diff --git a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md index 768830147..0fb82f8e8 100644 --- a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md +++ b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md @@ -16,29 +16,49 @@ This document provides detailed guidance for migrating from Elasticsearch (ES) t ## Migration Strategy Overview +There are two migration mechanisms. Choose the mechanism first, then follow the matching section. + | Source Version | Target Version | Migration Method | Notes | | :--- | :--- | :--- | :--- | | **ES 7.10** | **OS 2.x** | Snapshot & Restore | ✅ Direct restore supported | -| **ES 7.10** | **OS 3.x** | Snapshot & Restore → Upgrade | ⚠️ Must restore to OS 2.x first, then upgrade | +| **ES 7.10** | **OS 3.x** | Snapshot & Restore → Reindex → Upgrade | ⚠️ Must restore to OS 2.x first, then upgrade | +| **ES 7.10** | **OS 3.x** | Reindex from Remote | ✅ Single step, no intermediate 2.x cluster | | **ES 8.x** | **OS 3.x** | Reindex from Remote | ✅ Direct migration supported | +### Choosing a Method + +**Snapshot & Restore** copies the index files themselves. It is much faster for large datasets and it +preserves index settings, mappings and aliases exactly. Its limitation is version compatibility: +OpenSearch 3.x can only open indices created by OpenSearch 2.0.0 or later, so an ES 7.10 index has to +be landed on OpenSearch 2.x and reindexed there before the cluster is upgraded. It also requires a +snapshot repository that both clusters can reach. + +**Reindex from Remote** reads documents over HTTP from the source cluster and writes them as new +documents on the target. Because every document is indexed afresh, the source's file format never +matters — this is why it can go from ES 7.10 straight to OpenSearch 3.x, and why it is the only +option for ES 8.x. The costs are that it pays the full indexing cost for every document (far slower +than restoring files at scale), it requires network connectivity from OpenSearch to the source, and +it copies **only documents** — index settings, mappings and aliases are not carried over and must be +created on the target beforehand. + :::warning Key Compatibility Note -- **ES 7.10 → OS 3.x direct restore is NOT supported**. OpenSearch 3.x requires indices to be created with OpenSearch 2.0.0+. -- ES 7.10 snapshots must be restored to OpenSearch 2.x first, then upgrade the cluster to OS 3.x. -- ES 8.x uses incompatible Lucene versions, so Snapshot & Restore is not available; use Reindex from Remote instead. +- **ES 7.10 → OS 3.x direct restore is NOT supported**. OpenSearch 3.x requires indices to be created with OpenSearch 2.0.0+. Attempting it fails with `snapshot_restore_exception: cannot restore index ... because it cannot be upgraded`. +- ES 7.10 snapshots must be restored to OpenSearch 2.x first, reindexed there, and only then upgraded to OS 3.x. Reindex from Remote avoids this entirely. +- **OpenSearch cannot restore snapshots taken by Elasticsearch 8.x.** Use Reindex from Remote for an ES 8.x source. ::: This guide uses ES 7.10 as the source for the Snapshot & Restore method, and ES 8.17 for the Reindex from Remote method. Adjust accordingly if your source version differs. -## Migrate from ES 7.10 to OpenSearch 2.x to 3.x +## Migrate from ES 7.10 to OpenSearch 3.x (via 2.x) -This migration requires a **two-phase approach**: +This migration requires a **three-phase approach**: -* **Phase 1**: Restore ES 7.10 snapshot to OpenSearch 2.x -* **Phase 2**: Upgrade OpenSearch 2.x to 3.x +* **Phase 0**: Create a snapshot on the source Elasticsearch 7.10 cluster +* **Phase 1**: Restore that snapshot to OpenSearch 2.x +* **Phase 2**: Reindex the restored indices, then upgrade OpenSearch 2.x to 3.x ### Prerequisites @@ -47,28 +67,46 @@ This migration requires a **two-phase approach**: #### Check if Plugin is Installed +Run the check on both clusters: + ```bash +# On an Elasticsearch pod curl -u "elastic:" "http://localhost:9200/_cat/plugins?v" + +# On an OpenSearch pod +curl -k -u "admin:" "https://localhost:9200/_cat/plugins?v" ``` :::info -Remember to replace `` with your cluster's credentials above and in the following commands. +Remember to replace `` with your cluster's credentials, here and in every command that follows. -For Elasticsearch, the default user is `elastic`, and the default password is randomly generated during creation. +- For Elasticsearch, the default user is `elastic`, and the password is randomly generated during creation. +- For OpenSearch, the default user is `admin`, and the password is stored in the `-admin-password` Secret. See [How to Set and Update the OpenSearch Admin Password](./How_to_update_opensearch_admin_password.md) for details. ::: #### Install repository-s3 Plugin -Plugin download URLs: +First, read the exact version each cluster runs — you need it to build the download URL: + +```bash +# On an Elasticsearch pod +curl -s -u "elastic:" "http://localhost:9200" | grep '"number"' -| Version | Download URL | +# On an OpenSearch pod +curl -sk -u "admin:" "https://localhost:9200" | grep '"number"' +``` + +Then substitute that version into the matching URL pattern: + +| Product | Download URL pattern | | :--- | :--- | -| ES 7.10.2 | `https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.10.2.zip` | -| OpenSearch 2.19.3 | `https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip` | -| OpenSearch 3.3.1 | `https://artifacts.opensearch.org/releases/plugins/repository-s3/3.3.1/repository-s3-3.3.1.zip` | +| Elasticsearch | `https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-.zip` | +| OpenSearch | `https://artifacts.opensearch.org/releases/plugins/repository-s3//repository-s3-.zip` | -:::note -The plugin version **must exactly match** the Elasticsearch/OpenSearch version. For example, OpenSearch 3.3.1 requires `repository-s3-3.3.1.zip`. +:::warning Version must match exactly +`elasticsearch-plugin install` / `opensearch-plugin install` reads the version recorded in the plugin package and **refuses to install it on a node running any other version** — including a different patch release. A mismatch leaves the node unable to start. + +The examples in this document use `7.10.2` for Elasticsearch and `2.19.3` / `3.3.1` for OpenSearch. These are examples only: **replace every occurrence with the versions your own clusters actually run.** ::: :::warning Air-Gapped Environments @@ -85,19 +123,19 @@ In **Application Container Platform** > **Applications** > **Applications** page - Click **Update** - Switch to **YAML** edit page -Update `values.yaml` on the **Custom** input textarea with the following content": +Update `values.yaml` on the **Custom** input textarea with the following content: ```yaml masterNodes: config: elasticsearch.yml: | - s3.client.default.endpoint: "" + s3.client.default.endpoint: "http://minio.example.com:9000" s3.client.default.region: "us-east-1" s3.client.default.path_style_access: true # Required for MinIO extraInitContainers: - name: install-plugins - image: harbor.alauda.cn/middleware/elasticsearch:v7.10.2 + image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2 command: - sh - -c @@ -117,7 +155,9 @@ extraVolumeMounts: ``` :::note -The above configuration only sets S3 configs for master nodes. If you have dedicated data nodes, add the same S3 config to `dataNodes` as well. +- The init container must use the **same Elasticsearch image your nodes already run**. The image above is an example. +- Mounting an `emptyDir` at `/usr/share/elasticsearch/plugins` hides anything already installed in that directory, so the init container must install every plugin the cluster needs, not just `repository-s3`. +- The above configuration only sets S3 configs for master nodes. If you have dedicated data nodes, add the same S3 config to `dataNodes` as well. ::: **OpenSearch:** @@ -135,18 +175,22 @@ spec: - https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip general: additionalConfig: - s3.client.default.endpoint: "" + s3.client.default.endpoint: "http://minio.example.com:9000" s3.client.default.region: "us-east-1" s3.client.default.path_style_access: "true" pluginsList: - https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip ``` -:::note -Both approaches will trigger a rolling restart of nodes to load the newly installed plugin. +:::warning Every change to `additionalConfig` or `pluginsList` restarts the whole cluster + +- Both approaches trigger a **rolling restart of every node**, one at a time, to load the new configuration or plugin. +- Every entry under `additionalConfig` is rendered as an environment variable on **every** pod, including the transient bootstrap pod, and OpenSearch reads it as a setting. The Operator **does not validate these values**. An unknown or misspelled setting is only rejected when the node boots, and the node then fails to start — see [Troubleshooting](#troubleshooting). +- Because nodes are restarted one at a time, verify that the first restarted node returns to `Running` and `Ready` before letting the rollout continue. If it does not, fix the configuration before the remaining nodes pick it up. +- Plugins in `pluginsList` are downloaded and installed **on every pod start**, not once. The URL must stay reachable from every node, or use an image with the plugin pre-installed. ::: -### Procedure +### Phase 0: Create the Snapshot on Elasticsearch 7.10 #### Step 1: Configure S3 Credentials @@ -174,6 +218,21 @@ For security reasons, avoid including access keys directly in API request bodies curl -u "elastic:" -X POST "http://localhost:9200/_nodes/reload_secure_settings" ``` + :::warning Repeat on every Elasticsearch pod + The keystore is a file inside each node's own config directory, and `reload_secure_settings` only reloads what is already present on each node. **Run step 1 on every Elasticsearch pod** (masters and data nodes) before calling the reload. + + Two things make this easy to get wrong: + + - `reload_secure_settings` reports success for **every** node even when most of them have no credentials at all, so its output is not a check that the credentials are in place. + - The failure surfaces later, when the repository is registered, and it names the **elected master** — which is usually not the pod you ran the keystore commands in: + + ```text + repository_verification_exception: [migration_repo] path [es_710_backup] is not accessible on master node + ``` + + The keystore also lives in the container filesystem: unless your chart persists the Elasticsearch config directory, the credentials are lost when a pod restarts and must be added again. + ::: + **On OpenSearch:** Use the Operator's declarative configuration: @@ -230,21 +289,27 @@ curl -u "elastic:" -X PUT "http://localhost:9200/_snapshot/migration_r curl -u "elastic:" -X PUT "http://localhost:9200/_snapshot/migration_repo/snapshot_1?wait_for_completion=true" \ -H 'Content-Type: application/json' -d' { - "indices": "*", + "indices": "*,-.kibana*,-.security*,-.monitoring*,-apm*,-.apm*", "ignore_unavailable": true, - "include_global_state": true + "include_global_state": true }' ``` :::note Excluding System Indices -It is recommended to exclude system indices (`.kibana*`, `.security*`, `.monitoring*`, `apm*`, `.apm*`) during snapshot creation. These indices are Elasticsearch-specific and will conflict with OpenSearch's internal indices during restore. By excluding them at snapshot time, you reduce snapshot size and avoid potential restore issues. +The `indices` pattern above excludes system indices (`.kibana*`, `.security*`, `.monitoring*`, `apm*`, `.apm*`). These indices are Elasticsearch-specific and would conflict with OpenSearch's internal indices during restore. Excluding them at snapshot time also reduces the snapshot size. + +Because these indices are not migrated, the objects they hold do not come across either: Kibana saved objects (index patterns, visualizations, dashboards) and Elasticsearch users, roles and role mappings must be recreated on the OpenSearch side. ::: ### Phase 1: Restore to OpenSearch 2.x #### Step 1: Deploy OpenSearch 2.x Cluster -Deploy a new OpenSearch **2.x** cluster using the OpenSearch Operator: +Deploy a new OpenSearch **2.x** cluster using the OpenSearch Operator. For the full deployment procedure, see the [OpenSearch Installation Guide](./OpenSearch_Installation_Guide.md); the fragment below shows only the fields this migration needs. + +:::note +Set `version` to an OpenSearch version that is available in your environment. On a cluster without external network access, only the OpenSearch versions included in the installed plugin package can be pulled — check which ones are available before you deploy, and use that version in the `pluginsList` URL as well. +::: ```yaml apiVersion: opensearch.opster.io/v1 @@ -252,12 +317,19 @@ kind: OpenSearchCluster metadata: name: my-cluster spec: + bootstrap: + # REQUIRED: the bootstrap pod also receives general.additionalConfig, so it needs the + # plugin that defines the s3.client.* settings, or it will not start. + pluginsList: + - https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip general: version: 2.19.3 additionalConfig: - s3.client.default.endpoint: "" + s3.client.default.endpoint: "http://minio.example.com:9000" s3.client.default.region: "us-east-1" s3.client.default.path_style_access: "true" + # OpenSearch 2.12 and later refuse to start without an initial admin password. + OPENSEARCH_INITIAL_ADMIN_PASSWORD: "" pluginsList: - https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip keystore: @@ -271,8 +343,30 @@ spec: base_path: es_710_backup readonly: "true" ... + security: + config: + # credentials the Operator itself uses to reach the cluster; the password must be the + # same one set through OPENSEARCH_INITIAL_ADMIN_PASSWORD above + adminCredentialsSecret: + name: admin-credentials +``` + +:::warning Two settings the cluster will not start without + +**1. `bootstrap.pluginsList`.** The Operator renders every `general.additionalConfig` entry as an environment variable on **all** pods, including the transient bootstrap pod. The `s3.client.*` entries are only valid settings when `repository-s3` is installed, so if the bootstrap pod does not install the plugin it fails immediately with: + +```text +StartupException: unknown setting [s3.client.default.region] please check that any +required plugins are installed, or check the breaking changes documentation for removed settings ``` +The bootstrap pod then crash-loops, and because the other nodes are pinned to it through `cluster.initial_master_nodes`, the cluster never forms. + +**2. An initial admin password.** From OpenSearch 2.12 onwards the demo password is rejected and the node exits with `No custom admin password found. Please provide a password via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD`. `security.config.adminCredentialsSecret` does **not** supply it — that secret is only used by the Operator to authenticate to the cluster. Because the bootstrap pod has no `env` field of its own, the password has to be passed through `additionalConfig`, which the Operator turns into environment variables on every pod. + +Note that a value placed in `additionalConfig` is stored in plain text in the cluster resource. Restrict access to the resource accordingly, and prefer supplying a complete security configuration through `security.config.securityConfigSecret` for anything beyond a short-lived migration cluster. +::: + #### Step 2: Restore the Snapshot on OpenSearch Exclude system indices to avoid conflicts with OpenSearch's internal indices: @@ -286,10 +380,10 @@ curl -k -u "admin:" -X POST "https://localhost:9200/_snapshot/migratio }' ``` -:::info -Remember to replace `` with your cluster's credentials above and in the following commands. - -For OpenSearch, the default user is `admin`, and the default password is `admin`. +:::note +- A restore fails if an index of the same name already exists and is open. Delete or close the target index first, or use `rename_pattern` / `rename_replacement` to restore under a different name. +- Restored indices keep the replica count of the source cluster. If the target cluster has fewer nodes, add `"index_settings": {"index.number_of_replicas": 1}` to the request body, otherwise the restored indices stay yellow. +- `include_global_state` is `false`, so index templates, legacy templates and ingest pipelines are **not** restored. Recreate the ones you need on OpenSearch. Index Lifecycle Management (ILM) policies have no direct equivalent and must be rebuilt as Index State Management (ISM) policies. ::: #### Step 3: Verification @@ -315,20 +409,28 @@ Indices restored from ES 7.10 snapshots retain their original version metadata ( For each restored index, create a new index and reindex the data: :::note -The examples below use `migration_test` as the index name. Replace `migration_test` with your actual index name when executing these commands. +- The examples below use `migration_test` as the index name. Replace `migration_test` with your actual index name when executing these commands. +- The commands require `jq`. If it is not available in the OpenSearch container, run them from a workstation that can reach the cluster. +- Copying the index **settings** matters: shard counts, custom analyzers and similar settings live there, not in the mappings. If an index uses a custom analyzer, the corresponding analysis plugin must also be installed on the target cluster before the new index can be created. ::: ```bash -# 1. Get the original index mapping and extract the mappings object using sed +# 1. Export the source index definition (settings AND mappings), removing the +# read-only fields that cannot be set on a new index -curl -s -k -u "admin:" "https://localhost:9200/migration_test/_mapping" | \ - sed 's/^{"migration_test"://' | sed 's/}$//' > mapping.json +curl -s -k -u "admin:" "https://localhost:9200/migration_test" | \ + jq '.migration_test + | {settings: .settings, mappings: .mappings} + | del(.settings.index.uuid, + .settings.index.creation_date, + .settings.index.version, + .settings.index.provided_name)' > index_def.json -# 2. Create a new index with the same mapping (add suffix _v2) +# 2. Create a new index with the same settings and mappings (add suffix _v2) curl -k -u "admin:" -X PUT "https://localhost:9200/migration_test_v2" \ -H 'Content-Type: application/json' \ - -d @mapping.json + -d @index_def.json # 3. Reindex data from old index to new index @@ -356,11 +458,15 @@ Repeat for all restored indices. After reindexing, verify the new index version: curl -k -u "admin:" "https://localhost:9200/migration_test_v2/_settings?filter_path=**.version" ``` -The `version.created` should show an OpenSearch 2.x internal version number (e.g., `136408127` for OS 2.19.x). ES 7.10.2 indices show `7102099`. If you see a number starting with `136` or higher, the reindex was successful. +The `version.created` should show an OpenSearch 2.x internal version number (for example `136408427` for OS 2.19.6), rather than the `7100299` that ES 7.10.2 indices carry. The exact number varies with the patch release, so do not compare against a literal: any `136xxxxxx` value means the index was created by OpenSearch 2.x and the reindex was successful. #### Step 2: Upgrade OpenSearch Cluster -Update the `OpenSearchCluster` CR to upgrade the version: +:::warning +Take a snapshot of the OpenSearch 2.x cluster before starting the upgrade. A major version upgrade cannot be rolled back in place. +::: + +Update the `OpenSearchCluster` CR to upgrade the version. Use the same version for OpenSearch and OpenSearch Dashboards: ```yaml spec: @@ -369,7 +475,7 @@ spec: pluginsList: - https://artifacts.opensearch.org/releases/plugins/repository-s3/3.3.1/repository-s3-3.3.1.zip dashboards: - version: 3.3.0 # Upgrade OpenSearch Dashboards as well + version: 3.3.1 # Upgrade OpenSearch Dashboards to the matching version ``` The Operator will perform a rolling upgrade automatically. @@ -385,11 +491,14 @@ curl -k -u "admin:" "https://localhost:9200/_cluster/health?pretty" ## Migrate from ES 8.x to OpenSearch 3.x -Elasticsearch 8.x uses a newer Lucene version with incompatible metadata protocols, making snapshots unreadable by OpenSearch. Use **Reindex from Remote** instead. +OpenSearch cannot restore snapshots taken by Elasticsearch 8.x, so **Reindex from Remote** is the only available method for this source version. + +The same method also works for an ES 7.10 source and, unlike Snapshot & Restore, can target OpenSearch 3.x directly without an intermediate 2.x cluster. See [Choosing a Method](#choosing-a-method) for the trade-offs. ### Prerequisites -- **Network Connectivity**: The OpenSearch cluster must be able to reach the ES 8.x cluster's HTTP/REST port (typically 9200). +- **Network Connectivity**: The OpenSearch cluster must be able to reach the source cluster's HTTP/REST port (typically 9200). +- **Index settings and mappings**: reindex copies documents only. Create the target index with the settings and mappings you need **before** reindexing, or the target index is created from dynamic mapping defaults and will not reproduce the source's shard count, custom analyzers or field types. ### Deploy ES 8.x Using ECK Operator @@ -456,13 +565,23 @@ Add the following configurations to `OpenSearchCluster` CR's `additionalConfig`: spec: general: additionalConfig: - # Allow connections to ES 8.x host (OpenSearch 3.x uses 'allowlist') + # Allow connections to the ES 8.x host. Host and port only - no http:// or https:// prefix. + # Separate multiple hosts with commas. reindex.remote.allowlist: "es8-cluster-host:9200" # Disable SSL verification for self-signed certificates reindex.ssl.verification_mode: "none" ``` -> **Note**: Nodes will be restarted after applying this configuration change. +:::warning It is `allowlist`, not `whitelist` + +Elasticsearch, and OpenSearch 1.x, used `reindex.remote.whitelist`. OpenSearch renamed the setting to `reindex.remote.allowlist` and kept the old name as a deprecated alias in 2.x. **OpenSearch 3.x removed the old name entirely**, so a configuration copied from Elasticsearch documentation makes every node fail at startup with: + +```text +SettingsException[unknown setting [reindex.remote.whitelist] ...] +``` + +Applying this change restarts the nodes one at a time. Confirm the first restarted node returns to `Running` and `Ready` before the rollout continues — see [Troubleshooting](#troubleshooting). +::: #### Step 2: Create Index Templates on OpenSearch (Optional but Recommended) @@ -517,6 +636,48 @@ curl -k -u "admin:" "https://localhost:9200/migration_test/_count" curl -k -u "elastic:" "https://es8-cluster-host:9200/migration_test/_count" ``` +## Troubleshooting + +### A node stays in CrashLoopBackOff after a configuration change + +Every entry under `spec.general.additionalConfig` is rendered as an environment variable on the pods and read by OpenSearch as a setting. The Operator does not validate these values. An unknown or misspelled setting is rejected when the node boots, and the node never starts: + +```text +[ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] uncaught exception in thread [main] +org.opensearch.bootstrap.StartupException: SettingsException[unknown setting [reindex.remote.whitelist] + please check that any required plugins are installed, or check the breaking changes documentation + for removed settings] +``` + +The Operator restarts nodes one at a time and waits for each one to become ready, so the rollout stops at the first node that fails. The remaining nodes keep running the previous configuration, which is why the cluster can still be serving traffic while one pod restarts in a loop. + +To recover: + +```bash +# 1. Identify the failing node and the rejected setting +kubectl get pods -n +kubectl logs -n -masters-0 --tail=50 + +# 2. Correct the setting in the cluster resource +kubectl edit opensearchcluster -n + +# 3. Confirm the Operator has regenerated the configuration. +# additionalConfig entries become environment variables on the pods - the Operator does +# not write an opensearch.yml ConfigMap, so check the StatefulSet, not a ConfigMap. +kubectl get sts -n - \ + -o jsonpath='{.spec.template.spec.containers[0].env}' | tr ',' '\n' | grep '' + +# 4. Restart the failing pod so it picks up the new configuration +kubectl delete pod -n -masters-0 + +# 5. Watch the rollout continue to the remaining nodes +kubectl get pods -n -w +``` + +:::note +Settings are validated one at a time, so the node reports only the first invalid setting it finds. If it fails again after your fix, repeat the procedure for the next reported setting. +::: + ## Client Migration Guide Regardless of the source ES version, **it is strongly recommended to switch to the official OpenSearch clients**. diff --git a/docs/zh/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md b/docs/zh/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md index de590180f..6ade691c9 100644 --- a/docs/zh/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md +++ b/docs/zh/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md @@ -33,12 +33,13 @@ sourceSHA: 4d71480f5d5df1aa4044adeeedaadb443b8cd9b2bd075df00f762f1ec153febe 本指南使用 ES 7.10 作为快照与恢复方法的源,使用 ES 8.17 作为从远程重建索引的方法。如果您的源版本不同,请相应调整。 -## 从 ES 7.10 迁移到 OpenSearch 2.x 到 3.x +## 从 ES 7.10 迁移到 OpenSearch 3.x(经由 2.x) -此迁移需要 **两阶段方法**: +此迁移需要 **三阶段方法**: -- **第一阶段**:将 ES 7.10 快照恢复到 OpenSearch 2.x -- **第二阶段**:将 OpenSearch 2.x 升级到 3.x +- **第 0 阶段**:在源 Elasticsearch 7.10 集群上创建快照 +- **第一阶段**:将该快照恢复到 OpenSearch 2.x +- **第二阶段**:对恢复的索引重建索引,然后将 OpenSearch 2.x 升级到 3.x ### 前提条件 @@ -47,28 +48,46 @@ sourceSHA: 4d71480f5d5df1aa4044adeeedaadb443b8cd9b2bd075df00f762f1ec153febe #### 检查插件是否已安装 +在两个集群上分别执行检查: + ```bash +# 在 Elasticsearch pod 上 curl -u "elastic:" "http://localhost:9200/_cat/plugins?v" + +# 在 OpenSearch pod 上 +curl -k -u "admin:" "https://localhost:9200/_cat/plugins?v" ``` :::info -请记得在上述命令和以下命令中将 `` 替换为您集群的凭据。 +请记得在此处以及后续所有命令中,将 `` 替换为您集群的凭据。 -对于 Elasticsearch,默认用户为 `elastic`,默认密码在创建时随机生成。 +- 对于 Elasticsearch,默认用户为 `elastic`,密码在创建时随机生成。 +- 对于 OpenSearch,默认用户为 `admin`,密码保存在 `<集群名称>-admin-password` Secret 中。详见 [如何设置和更新 OpenSearch 管理员密码](./How_to_update_opensearch_admin_password.md)。 ::: #### 安装 repository-s3 插件 -插件下载 URL: +首先读取每个集群实际运行的版本号,下载 URL 需要使用它: + +```bash +# 在 Elasticsearch pod 上 +curl -s -u "elastic:" "http://localhost:9200" | grep '"number"' -| 版本 | 下载 URL | -| :--------------- | :-------------------------------------------------------------------------------------------------- | -| ES 7.10.2 | `https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.10.2.zip` | -| OpenSearch 2.19.3 | `https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip` | -| OpenSearch 3.3.1 | `https://artifacts.opensearch.org/releases/plugins/repository-s3/3.3.1/repository-s3-3.3.1.zip` | +# 在 OpenSearch pod 上 +curl -sk -u "admin:" "https://localhost:9200" | grep '"number"' +``` -:::note -插件版本 **必须完全匹配** Elasticsearch/OpenSearch 版本。例如,OpenSearch 3.3.1 需要 `repository-s3-3.3.1.zip`。 +然后将该版本号代入对应的 URL 模板: + +| 产品 | 下载 URL 模板 | +| :--- | :--- | +| Elasticsearch | `https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-.zip` | +| OpenSearch | `https://artifacts.opensearch.org/releases/plugins/repository-s3//repository-s3-.zip` | + +:::warning 版本必须完全一致 +`elasticsearch-plugin install` / `opensearch-plugin install` 会读取插件包中记录的版本,并 **拒绝安装到任何其他版本的节点上**(补丁版本不同也不行)。版本不匹配会导致节点无法启动。 + +本文档示例中 Elasticsearch 使用 `7.10.2`,OpenSearch 使用 `2.19.3` / `3.3.1`。这些仅为示例:**请将文中每一处都替换为您集群实际运行的版本。** ::: :::warning 空气隔离环境 @@ -91,13 +110,13 @@ curl -u "elastic:" "http://localhost:9200/_cat/plugins?v" masterNodes: config: elasticsearch.yml: | - s3.client.default.endpoint: "" + s3.client.default.endpoint: "http://minio.example.com:9000" s3.client.default.region: "us-east-1" s3.client.default.path_style_access: true # MinIO 所需 extraInitContainers: - name: install-plugins - image: harbor.alauda.cn/middleware/elasticsearch:v7.10.2 + image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2 command: - sh - -c @@ -117,7 +136,9 @@ extraVolumeMounts: ``` :::note -上述配置仅为主节点设置 S3 配置。如果您有专用的数据节点,请将相同的 S3 配置添加到 `dataNodes` 中。 +- init 容器必须使用 **与您节点当前运行的相同的 Elasticsearch 镜像**,上面的镜像仅为示例。 +- 在 `/usr/share/elasticsearch/plugins` 上挂载 `emptyDir` 会遮蔽该目录中已安装的内容,因此 init 容器必须安装集群需要的全部插件,而不仅仅是 `repository-s3`。 +- 上述配置仅为主节点设置 S3 配置。如果您有专用的数据节点,请将相同的 S3 配置添加到 `dataNodes` 中。 ::: **OpenSearch:** @@ -135,18 +156,22 @@ spec: - https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip general: additionalConfig: - s3.client.default.endpoint: "" + s3.client.default.endpoint: "http://minio.example.com:9000" s3.client.default.region: "us-east-1" s3.client.default.path_style_access: "true" pluginsList: - https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip ``` -:::note -这两种方法都会触发节点的滚动重启,以加载新安装的插件。 +:::warning 每次修改 `additionalConfig` 或 `pluginsList` 都会重启整个集群 + +- 这两种方法都会触发 **所有节点逐个滚动重启**,以加载新的配置或插件。 +- `additionalConfig` 下的值会被直接写入 `opensearch.yml`,Operator **不会对其做校验**。未知或拼写错误的设置只有在节点启动时才会被拒绝,此时节点将无法启动 —— 参见 [故障排查](#故障排查)。 +- 由于节点是逐个重启的,请先确认第一个重启的节点恢复到 `Running` 且 `Ready`,再让滚动继续。如果没有恢复,请在其余节点应用该配置之前先修正配置。 +- `pluginsList` 中的插件会在 **每次 Pod 启动时** 下载并安装,而不是只装一次。该 URL 必须对每个节点持续可达,或改用已预装插件的镜像。 ::: -### 操作步骤 +### 第 0 阶段:在 Elasticsearch 7.10 上创建快照 #### 第 1 步:配置 S3 凭据 @@ -174,6 +199,12 @@ spec: curl -u "elastic:" -X POST "http://localhost:9200/_nodes/reload_secure_settings" ``` + :::warning 每个 Elasticsearch Pod 都要执行 + 密钥库是每个节点自己配置目录中的文件,而 `reload_secure_settings` 只会重新加载各节点上已经存在的内容。**请在每一个 Elasticsearch Pod(主节点和数据节点)上都执行第 1 步**,然后再调用重新加载,否则缺少凭据的节点会导致快照失败。 + + 密钥库同样位于容器文件系统中:除非您的 Chart 对 Elasticsearch 配置目录做了持久化,否则 Pod 重启后凭据会丢失,需要重新添加。 + ::: + **在 OpenSearch 上:** 使用 Operator 的声明性配置: @@ -230,21 +261,27 @@ curl -u "elastic:" -X PUT "http://localhost:9200/_snapshot/migration_r curl -u "elastic:" -X PUT "http://localhost:9200/_snapshot/migration_repo/snapshot_1?wait_for_completion=true" \ -H 'Content-Type: application/json' -d' { - "indices": "*", + "indices": "*,-.kibana*,-.security*,-.monitoring*,-apm*,-.apm*", "ignore_unavailable": true, - "include_global_state": true + "include_global_state": true }' ``` :::note 排除系统索引 -建议在创建快照时排除系统索引(`.kibana*`、`.security*`、`.monitoring*`、`apm*`、`.apm*`)。这些索引是 Elasticsearch 特有的,在恢复时会与 OpenSearch 的内部索引发生冲突。通过在快照时排除它们,可以减少快照大小并避免潜在的恢复问题。 +上面的 `indices` 模式已排除系统索引(`.kibana*`、`.security*`、`.monitoring*`、`apm*`、`.apm*`)。这些索引是 Elasticsearch 特有的,在恢复时会与 OpenSearch 的内部索引发生冲突;在快照时排除它们还可以减小快照体积。 + +由于这些索引不会被迁移,它们所保存的对象也不会随之迁移:Kibana 的已保存对象(索引模式、可视化、仪表板)以及 Elasticsearch 的用户、角色和角色映射,都需要在 OpenSearch 侧重新创建。 ::: ### 第一阶段:恢复到 OpenSearch 2.x #### 第 1 步:部署 OpenSearch 2.x 集群 -使用 OpenSearch Operator 部署一个新的 OpenSearch **2.x** 集群: +使用 OpenSearch Operator 部署一个新的 OpenSearch **2.x** 集群。完整的部署流程请参见 [OpenSearch 安装指南](./OpenSearch_Installation_Guide.md);下面的片段只列出本次迁移需要的字段。 + +:::note +请将 `version` 设置为您环境中可用的 OpenSearch 版本。在没有外网访问的集群上,只有已安装插件包中包含的 OpenSearch 版本才能拉取到镜像 —— 请先确认哪些版本可用,并在 `pluginsList` 的 URL 中使用同一版本。 +::: ```yaml apiVersion: opensearch.opster.io/v1 @@ -255,7 +292,7 @@ spec: general: version: 2.19.3 additionalConfig: - s3.client.default.endpoint: "" + s3.client.default.endpoint: "http://minio.example.com:9000" s3.client.default.region: "us-east-1" s3.client.default.path_style_access: "true" pluginsList: @@ -286,10 +323,10 @@ curl -k -u "admin:" -X POST "https://localhost:9200/_snapshot/migratio }' ``` -:::info -请记得在上述命令和以下命令中将 `` 替换为您集群的凭据。 - -对于 OpenSearch,默认用户为 `admin`,默认密码为 `admin`。 +:::note +- 如果同名索引已存在且处于 open 状态,恢复会失败。请先删除或关闭目标索引,或使用 `rename_pattern` / `rename_replacement` 以其他名称恢复。 +- 恢复出来的索引会保留源集群的副本数。如果目标集群节点更少,请在请求体中加入 `"index_settings": {"index.number_of_replicas": 1}`,否则恢复后的索引会一直是 yellow 状态。 +- `include_global_state` 为 `false`,因此索引模板、旧版模板和 ingest pipeline **不会** 被恢复,需要在 OpenSearch 上重新创建所需的部分。索引生命周期管理(ILM)策略没有直接对应物,必须重建为索引状态管理(ISM)策略。 ::: #### 第 3 步:验证 @@ -315,20 +352,27 @@ curl -k -u "admin:" "https://localhost:9200//_count" 对于每个恢复的索引,创建一个新索引并重建数据: :::note -以下示例使用 `migration_test` 作为索引名称。在执行这些命令时,请将 `migration_test` 替换为您的实际索引名称。 +- 以下示例使用 `migration_test` 作为索引名称。在执行这些命令时,请将 `migration_test` 替换为您的实际索引名称。 +- 这些命令需要 `jq`。如果 OpenSearch 容器中没有该工具,请在能够访问集群的工作机上执行。 +- 复制索引 **settings** 很重要:分片数、自定义分析器等配置保存在 settings 中,而不在 mappings 中。如果索引使用了自定义分析器,还必须先在目标集群上安装对应的分析插件,新索引才能创建成功。 ::: ```bash -# 1. 获取原始索引映射并使用 sed 提取映射对象 +# 1. 导出源索引定义(settings 和 mappings),并去掉新索引上无法设置的只读字段 -curl -s -k -u "admin:" "https://localhost:9200/migration_test/_mapping" | \ - sed 's/^{"migration_test"://' | sed 's/}$//' > mapping.json +curl -s -k -u "admin:" "https://localhost:9200/migration_test" | \ + jq '.migration_test + | {settings: .settings, mappings: .mappings} + | del(.settings.index.uuid, + .settings.index.creation_date, + .settings.index.version, + .settings.index.provided_name)' > index_def.json -# 2. 创建一个具有相同映射的新索引(添加后缀 _v2) +# 2. 使用相同的 settings 和 mappings 创建新索引(添加后缀 _v2) curl -k -u "admin:" -X PUT "https://localhost:9200/migration_test_v2" \ -H 'Content-Type: application/json' \ - -d @mapping.json + -d @index_def.json # 3. 从旧索引重建数据到新索引 @@ -356,11 +400,15 @@ curl -k -u "admin:" -X POST "https://localhost:9200/_aliases" \ curl -k -u "admin:" "https://localhost:9200/migration_test_v2/_settings?filter_path=**.version" ``` -`version.created` 应显示 OpenSearch 2.x 内部版本号(例如,`136408127` 对于 OS 2.19.x)。ES 7.10.2 索引显示 `7102099`。如果您看到以 `136` 开头或更高的数字,则重建成功。 +`version.created` 应显示 OpenSearch 2.x 的内部版本号(例如 OS 2.19.x 为 `136408127`),而不是 ES 7.10.2 索引所带的 `7102099`。任何 `136xxxxxx` 形式的值都表示该索引由 OpenSearch 2.x 创建,重建成功。 #### 第 2 步:升级 OpenSearch 集群 -更新 `OpenSearchCluster` CR 以升级版本: +:::warning +开始升级前,请先对 OpenSearch 2.x 集群做一次快照。大版本升级无法就地回滚。 +::: + +更新 `OpenSearchCluster` CR 以升级版本。OpenSearch 与 OpenSearch Dashboards 请使用相同的版本: ```yaml spec: @@ -369,7 +417,7 @@ spec: pluginsList: - https://artifacts.opensearch.org/releases/plugins/repository-s3/3.3.1/repository-s3-3.3.1.zip dashboards: - version: 3.3.0 # 同时升级 OpenSearch Dashboards + version: 3.3.1 # 将 OpenSearch Dashboards 升级到相同版本 ``` Operator 将自动执行滚动升级。 @@ -456,13 +504,23 @@ spec: spec: general: additionalConfig: - # 允许连接到 ES 8.x 主机(OpenSearch 3.x 使用 'allowlist') + # 允许连接到 ES 8.x 主机。只写主机和端口,不带 http:// 或 https:// 前缀。 + # 多个主机之间用逗号分隔。 reindex.remote.allowlist: "es8-cluster-host:9200" # 禁用自签名证书的 SSL 验证 reindex.ssl.verification_mode: "none" ``` -> **注意**:应用此配置更改后,节点将重启。 +:::warning 是 `allowlist`,不是 `whitelist` + +Elasticsearch 以及 OpenSearch 1.x 使用的是 `reindex.remote.whitelist`。OpenSearch 将该设置重命名为 `reindex.remote.allowlist`,并在 2.x 中把旧名称保留为已废弃的别名。**OpenSearch 3.x 彻底移除了旧名称**,因此从 Elasticsearch 文档照搬过来的配置会让每个节点在启动时失败: + +```text +SettingsException[unknown setting [reindex.remote.whitelist] ...] +``` + +应用该变更会让节点逐个重启。请确认第一个重启的节点恢复到 `Running` 且 `Ready`,再让滚动继续 —— 参见 [故障排查](#故障排查)。 +::: #### 第 2 步:在 OpenSearch 上创建索引模板(可选但推荐) @@ -517,6 +575,45 @@ curl -k -u "admin:" "https://localhost:9200/migration_test/_count" curl -k -u "elastic:" "https://es8-cluster-host:9200/migration_test/_count" ``` +## 故障排查 + +### 修改配置后节点一直处于 CrashLoopBackOff + +放在 `spec.general.additionalConfig` 下的值会被直接写入 `opensearch.yml`,Operator 不会对其做校验。未知或拼写错误的设置会在节点启动时被拒绝,节点将无法启动: + +```text +[ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] uncaught exception in thread [main] +org.opensearch.bootstrap.StartupException: SettingsException[unknown setting [reindex.remote.whitelist] + please check that any required plugins are installed, or check the breaking changes documentation + for removed settings] +``` + +Operator 逐个重启节点,并等待每个节点就绪,因此滚动会停在第一个失败的节点上。其余节点仍运行着此前的配置 —— 这正是为什么某个 Pod 反复重启时,集群仍可能在正常提供服务。 + +恢复步骤: + +```bash +# 1. 确认失败的节点以及被拒绝的设置 +kubectl get pods -n +kubectl logs -n -masters-0 --tail=50 + +# 2. 在集群资源中修正该设置 +kubectl edit opensearchcluster -n + +# 3. 确认 Operator 已重新生成配置 +kubectl get cm -n -o yaml | grep -n '' + +# 4. 重启失败的 Pod,使其加载新配置 +kubectl delete pod -n -masters-0 + +# 5. 观察滚动继续应用到其余节点 +kubectl get pods -n -w +``` + +:::note +设置是逐条校验的,节点只会报告它发现的第一个非法设置。如果修复后仍然启动失败,请针对下一个报告出来的设置重复上述步骤。 +::: + ## 客户端迁移指南 无论源 ES 版本如何,**强烈建议切换到官方 OpenSearch 客户端**。