diff --git a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md index 768830147..0fb82f8e8 100644 --- a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md +++ b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md @@ -16,29 +16,49 @@ This document provides detailed guidance for migrating from Elasticsearch (ES) t ## Migration Strategy Overview +There are two migration mechanisms. Choose the mechanism first, then follow the matching section. + | Source Version | Target Version | Migration Method | Notes | | :--- | :--- | :--- | :--- | | **ES 7.10** | **OS 2.x** | Snapshot & Restore | ✅ Direct restore supported | -| **ES 7.10** | **OS 3.x** | Snapshot & Restore → Upgrade | ⚠️ Must restore to OS 2.x first, then upgrade | +| **ES 7.10** | **OS 3.x** | Snapshot & Restore → Reindex → Upgrade | ⚠️ Must restore to OS 2.x first, then upgrade | +| **ES 7.10** | **OS 3.x** | Reindex from Remote | ✅ Single step, no intermediate 2.x cluster | | **ES 8.x** | **OS 3.x** | Reindex from Remote | ✅ Direct migration supported | +### Choosing a Method + +**Snapshot & Restore** copies the index files themselves. It is much faster for large datasets and it +preserves index settings, mappings and aliases exactly. Its limitation is version compatibility: +OpenSearch 3.x can only open indices created by OpenSearch 2.0.0 or later, so an ES 7.10 index has to +be landed on OpenSearch 2.x and reindexed there before the cluster is upgraded. It also requires a +snapshot repository that both clusters can reach. + +**Reindex from Remote** reads documents over HTTP from the source cluster and writes them as new +documents on the target. Because every document is indexed afresh, the source's file format never +matters — this is why it can go from ES 7.10 straight to OpenSearch 3.x, and why it is the only +option for ES 8.x. The costs are that it pays the full indexing cost for every document (far slower +than restoring files at scale), it requires network connectivity from OpenSearch to the source, and +it copies **only documents** — index settings, mappings and aliases are not carried over and must be +created on the target beforehand. + :::warning Key Compatibility Note -- **ES 7.10 → OS 3.x direct restore is NOT supported**. OpenSearch 3.x requires indices to be created with OpenSearch 2.0.0+. -- ES 7.10 snapshots must be restored to OpenSearch 2.x first, then upgrade the cluster to OS 3.x. -- ES 8.x uses incompatible Lucene versions, so Snapshot & Restore is not available; use Reindex from Remote instead. +- **ES 7.10 → OS 3.x direct restore is NOT supported**. OpenSearch 3.x requires indices to be created with OpenSearch 2.0.0+. Attempting it fails with `snapshot_restore_exception: cannot restore index ... because it cannot be upgraded`. +- ES 7.10 snapshots must be restored to OpenSearch 2.x first, reindexed there, and only then upgraded to OS 3.x. Reindex from Remote avoids this entirely. +- **OpenSearch cannot restore snapshots taken by Elasticsearch 8.x.** Use Reindex from Remote for an ES 8.x source. ::: This guide uses ES 7.10 as the source for the Snapshot & Restore method, and ES 8.17 for the Reindex from Remote method. Adjust accordingly if your source version differs. -## Migrate from ES 7.10 to OpenSearch 2.x to 3.x +## Migrate from ES 7.10 to OpenSearch 3.x (via 2.x) -This migration requires a **two-phase approach**: +This migration requires a **three-phase approach**: -* **Phase 1**: Restore ES 7.10 snapshot to OpenSearch 2.x -* **Phase 2**: Upgrade OpenSearch 2.x to 3.x +* **Phase 0**: Create a snapshot on the source Elasticsearch 7.10 cluster +* **Phase 1**: Restore that snapshot to OpenSearch 2.x +* **Phase 2**: Reindex the restored indices, then upgrade OpenSearch 2.x to 3.x ### Prerequisites @@ -47,28 +67,46 @@ This migration requires a **two-phase approach**: #### Check if Plugin is Installed +Run the check on both clusters: + ```bash +# On an Elasticsearch pod curl -u "elastic:" "http://localhost:9200/_cat/plugins?v" + +# On an OpenSearch pod +curl -k -u "admin:" "https://localhost:9200/_cat/plugins?v" ``` :::info -Remember to replace `` with your cluster's credentials above and in the following commands. +Remember to replace `` with your cluster's credentials, here and in every command that follows. -For Elasticsearch, the default user is `elastic`, and the default password is randomly generated during creation. +- For Elasticsearch, the default user is `elastic`, and the password is randomly generated during creation. +- For OpenSearch, the default user is `admin`, and the password is stored in the `-admin-password` Secret. See [How to Set and Update the OpenSearch Admin Password](./How_to_update_opensearch_admin_password.md) for details. ::: #### Install repository-s3 Plugin -Plugin download URLs: +First, read the exact version each cluster runs — you need it to build the download URL: + +```bash +# On an Elasticsearch pod +curl -s -u "elastic:" "http://localhost:9200" | grep '"number"' -| Version | Download URL | +# On an OpenSearch pod +curl -sk -u "admin:" "https://localhost:9200" | grep '"number"' +``` + +Then substitute that version into the matching URL pattern: + +| Product | Download URL pattern | | :--- | :--- | -| ES 7.10.2 | `https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.10.2.zip` | -| OpenSearch 2.19.3 | `https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip` | -| OpenSearch 3.3.1 | `https://artifacts.opensearch.org/releases/plugins/repository-s3/3.3.1/repository-s3-3.3.1.zip` | +| Elasticsearch | `https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-.zip` | +| OpenSearch | `https://artifacts.opensearch.org/releases/plugins/repository-s3//repository-s3-.zip` | -:::note -The plugin version **must exactly match** the Elasticsearch/OpenSearch version. For example, OpenSearch 3.3.1 requires `repository-s3-3.3.1.zip`. +:::warning Version must match exactly +`elasticsearch-plugin install` / `opensearch-plugin install` reads the version recorded in the plugin package and **refuses to install it on a node running any other version** — including a different patch release. A mismatch leaves the node unable to start. + +The examples in this document use `7.10.2` for Elasticsearch and `2.19.3` / `3.3.1` for OpenSearch. These are examples only: **replace every occurrence with the versions your own clusters actually run.** ::: :::warning Air-Gapped Environments @@ -85,19 +123,19 @@ In **Application Container Platform** > **Applications** > **Applications** page - Click **Update** - Switch to **YAML** edit page -Update `values.yaml` on the **Custom** input textarea with the following content": +Update `values.yaml` on the **Custom** input textarea with the following content: ```yaml masterNodes: config: elasticsearch.yml: | - s3.client.default.endpoint: "" + s3.client.default.endpoint: "http://minio.example.com:9000" s3.client.default.region: "us-east-1" s3.client.default.path_style_access: true # Required for MinIO extraInitContainers: - name: install-plugins - image: harbor.alauda.cn/middleware/elasticsearch:v7.10.2 + image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2 command: - sh - -c @@ -117,7 +155,9 @@ extraVolumeMounts: ``` :::note -The above configuration only sets S3 configs for master nodes. If you have dedicated data nodes, add the same S3 config to `dataNodes` as well. +- The init container must use the **same Elasticsearch image your nodes already run**. The image above is an example. +- Mounting an `emptyDir` at `/usr/share/elasticsearch/plugins` hides anything already installed in that directory, so the init container must install every plugin the cluster needs, not just `repository-s3`. +- The above configuration only sets S3 configs for master nodes. If you have dedicated data nodes, add the same S3 config to `dataNodes` as well. ::: **OpenSearch:** @@ -135,18 +175,22 @@ spec: - https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip general: additionalConfig: - s3.client.default.endpoint: "" + s3.client.default.endpoint: "http://minio.example.com:9000" s3.client.default.region: "us-east-1" s3.client.default.path_style_access: "true" pluginsList: - https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip ``` -:::note -Both approaches will trigger a rolling restart of nodes to load the newly installed plugin. +:::warning Every change to `additionalConfig` or `pluginsList` restarts the whole cluster + +- Both approaches trigger a **rolling restart of every node**, one at a time, to load the new configuration or plugin. +- Every entry under `additionalConfig` is rendered as an environment variable on **every** pod, including the transient bootstrap pod, and OpenSearch reads it as a setting. The Operator **does not validate these values**. An unknown or misspelled setting is only rejected when the node boots, and the node then fails to start — see [Troubleshooting](#troubleshooting). +- Because nodes are restarted one at a time, verify that the first restarted node returns to `Running` and `Ready` before letting the rollout continue. If it does not, fix the configuration before the remaining nodes pick it up. +- Plugins in `pluginsList` are downloaded and installed **on every pod start**, not once. The URL must stay reachable from every node, or use an image with the plugin pre-installed. ::: -### Procedure +### Phase 0: Create the Snapshot on Elasticsearch 7.10 #### Step 1: Configure S3 Credentials @@ -174,6 +218,21 @@ For security reasons, avoid including access keys directly in API request bodies curl -u "elastic:" -X POST "http://localhost:9200/_nodes/reload_secure_settings" ``` + :::warning Repeat on every Elasticsearch pod + The keystore is a file inside each node's own config directory, and `reload_secure_settings` only reloads what is already present on each node. **Run step 1 on every Elasticsearch pod** (masters and data nodes) before calling the reload. + + Two things make this easy to get wrong: + + - `reload_secure_settings` reports success for **every** node even when most of them have no credentials at all, so its output is not a check that the credentials are in place. + - The failure surfaces later, when the repository is registered, and it names the **elected master** — which is usually not the pod you ran the keystore commands in: + + ```text + repository_verification_exception: [migration_repo] path [es_710_backup] is not accessible on master node + ``` + + The keystore also lives in the container filesystem: unless your chart persists the Elasticsearch config directory, the credentials are lost when a pod restarts and must be added again. + ::: + **On OpenSearch:** Use the Operator's declarative configuration: @@ -230,21 +289,27 @@ curl -u "elastic:" -X PUT "http://localhost:9200/_snapshot/migration_r curl -u "elastic:" -X PUT "http://localhost:9200/_snapshot/migration_repo/snapshot_1?wait_for_completion=true" \ -H 'Content-Type: application/json' -d' { - "indices": "*", + "indices": "*,-.kibana*,-.security*,-.monitoring*,-apm*,-.apm*", "ignore_unavailable": true, - "include_global_state": true + "include_global_state": true }' ``` :::note Excluding System Indices -It is recommended to exclude system indices (`.kibana*`, `.security*`, `.monitoring*`, `apm*`, `.apm*`) during snapshot creation. These indices are Elasticsearch-specific and will conflict with OpenSearch's internal indices during restore. By excluding them at snapshot time, you reduce snapshot size and avoid potential restore issues. +The `indices` pattern above excludes system indices (`.kibana*`, `.security*`, `.monitoring*`, `apm*`, `.apm*`). These indices are Elasticsearch-specific and would conflict with OpenSearch's internal indices during restore. Excluding them at snapshot time also reduces the snapshot size. + +Because these indices are not migrated, the objects they hold do not come across either: Kibana saved objects (index patterns, visualizations, dashboards) and Elasticsearch users, roles and role mappings must be recreated on the OpenSearch side. ::: ### Phase 1: Restore to OpenSearch 2.x #### Step 1: Deploy OpenSearch 2.x Cluster -Deploy a new OpenSearch **2.x** cluster using the OpenSearch Operator: +Deploy a new OpenSearch **2.x** cluster using the OpenSearch Operator. For the full deployment procedure, see the [OpenSearch Installation Guide](./OpenSearch_Installation_Guide.md); the fragment below shows only the fields this migration needs. + +:::note +Set `version` to an OpenSearch version that is available in your environment. On a cluster without external network access, only the OpenSearch versions included in the installed plugin package can be pulled — check which ones are available before you deploy, and use that version in the `pluginsList` URL as well. +::: ```yaml apiVersion: opensearch.opster.io/v1 @@ -252,12 +317,19 @@ kind: OpenSearchCluster metadata: name: my-cluster spec: + bootstrap: + # REQUIRED: the bootstrap pod also receives general.additionalConfig, so it needs the + # plugin that defines the s3.client.* settings, or it will not start. + pluginsList: + - https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip general: version: 2.19.3 additionalConfig: - s3.client.default.endpoint: "" + s3.client.default.endpoint: "http://minio.example.com:9000" s3.client.default.region: "us-east-1" s3.client.default.path_style_access: "true" + # OpenSearch 2.12 and later refuse to start without an initial admin password. + OPENSEARCH_INITIAL_ADMIN_PASSWORD: "" pluginsList: - https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip keystore: @@ -271,8 +343,30 @@ spec: base_path: es_710_backup readonly: "true" ... + security: + config: + # credentials the Operator itself uses to reach the cluster; the password must be the + # same one set through OPENSEARCH_INITIAL_ADMIN_PASSWORD above + adminCredentialsSecret: + name: admin-credentials +``` + +:::warning Two settings the cluster will not start without + +**1. `bootstrap.pluginsList`.** The Operator renders every `general.additionalConfig` entry as an environment variable on **all** pods, including the transient bootstrap pod. The `s3.client.*` entries are only valid settings when `repository-s3` is installed, so if the bootstrap pod does not install the plugin it fails immediately with: + +```text +StartupException: unknown setting [s3.client.default.region] please check that any +required plugins are installed, or check the breaking changes documentation for removed settings ``` +The bootstrap pod then crash-loops, and because the other nodes are pinned to it through `cluster.initial_master_nodes`, the cluster never forms. + +**2. An initial admin password.** From OpenSearch 2.12 onwards the demo password is rejected and the node exits with `No custom admin password found. Please provide a password via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD`. `security.config.adminCredentialsSecret` does **not** supply it — that secret is only used by the Operator to authenticate to the cluster. Because the bootstrap pod has no `env` field of its own, the password has to be passed through `additionalConfig`, which the Operator turns into environment variables on every pod. + +Note that a value placed in `additionalConfig` is stored in plain text in the cluster resource. Restrict access to the resource accordingly, and prefer supplying a complete security configuration through `security.config.securityConfigSecret` for anything beyond a short-lived migration cluster. +::: + #### Step 2: Restore the Snapshot on OpenSearch Exclude system indices to avoid conflicts with OpenSearch's internal indices: @@ -286,10 +380,10 @@ curl -k -u "admin:" -X POST "https://localhost:9200/_snapshot/migratio }' ``` -:::info -Remember to replace `` with your cluster's credentials above and in the following commands. - -For OpenSearch, the default user is `admin`, and the default password is `admin`. +:::note +- A restore fails if an index of the same name already exists and is open. Delete or close the target index first, or use `rename_pattern` / `rename_replacement` to restore under a different name. +- Restored indices keep the replica count of the source cluster. If the target cluster has fewer nodes, add `"index_settings": {"index.number_of_replicas": 1}` to the request body, otherwise the restored indices stay yellow. +- `include_global_state` is `false`, so index templates, legacy templates and ingest pipelines are **not** restored. Recreate the ones you need on OpenSearch. Index Lifecycle Management (ILM) policies have no direct equivalent and must be rebuilt as Index State Management (ISM) policies. ::: #### Step 3: Verification @@ -315,20 +409,28 @@ Indices restored from ES 7.10 snapshots retain their original version metadata ( For each restored index, create a new index and reindex the data: :::note -The examples below use `migration_test` as the index name. Replace `migration_test` with your actual index name when executing these commands. +- The examples below use `migration_test` as the index name. Replace `migration_test` with your actual index name when executing these commands. +- The commands require `jq`. If it is not available in the OpenSearch container, run them from a workstation that can reach the cluster. +- Copying the index **settings** matters: shard counts, custom analyzers and similar settings live there, not in the mappings. If an index uses a custom analyzer, the corresponding analysis plugin must also be installed on the target cluster before the new index can be created. ::: ```bash -# 1. Get the original index mapping and extract the mappings object using sed +# 1. Export the source index definition (settings AND mappings), removing the +# read-only fields that cannot be set on a new index -curl -s -k -u "admin:" "https://localhost:9200/migration_test/_mapping" | \ - sed 's/^{"migration_test"://' | sed 's/}$//' > mapping.json +curl -s -k -u "admin:" "https://localhost:9200/migration_test" | \ + jq '.migration_test + | {settings: .settings, mappings: .mappings} + | del(.settings.index.uuid, + .settings.index.creation_date, + .settings.index.version, + .settings.index.provided_name)' > index_def.json -# 2. Create a new index with the same mapping (add suffix _v2) +# 2. Create a new index with the same settings and mappings (add suffix _v2) curl -k -u "admin:" -X PUT "https://localhost:9200/migration_test_v2" \ -H 'Content-Type: application/json' \ - -d @mapping.json + -d @index_def.json # 3. Reindex data from old index to new index @@ -356,11 +458,15 @@ Repeat for all restored indices. After reindexing, verify the new index version: curl -k -u "admin:" "https://localhost:9200/migration_test_v2/_settings?filter_path=**.version" ``` -The `version.created` should show an OpenSearch 2.x internal version number (e.g., `136408127` for OS 2.19.x). ES 7.10.2 indices show `7102099`. If you see a number starting with `136` or higher, the reindex was successful. +The `version.created` should show an OpenSearch 2.x internal version number (for example `136408427` for OS 2.19.6), rather than the `7100299` that ES 7.10.2 indices carry. The exact number varies with the patch release, so do not compare against a literal: any `136xxxxxx` value means the index was created by OpenSearch 2.x and the reindex was successful. #### Step 2: Upgrade OpenSearch Cluster -Update the `OpenSearchCluster` CR to upgrade the version: +:::warning +Take a snapshot of the OpenSearch 2.x cluster before starting the upgrade. A major version upgrade cannot be rolled back in place. +::: + +Update the `OpenSearchCluster` CR to upgrade the version. Use the same version for OpenSearch and OpenSearch Dashboards: ```yaml spec: @@ -369,7 +475,7 @@ spec: pluginsList: - https://artifacts.opensearch.org/releases/plugins/repository-s3/3.3.1/repository-s3-3.3.1.zip dashboards: - version: 3.3.0 # Upgrade OpenSearch Dashboards as well + version: 3.3.1 # Upgrade OpenSearch Dashboards to the matching version ``` The Operator will perform a rolling upgrade automatically. @@ -385,11 +491,14 @@ curl -k -u "admin:" "https://localhost:9200/_cluster/health?pretty" ## Migrate from ES 8.x to OpenSearch 3.x -Elasticsearch 8.x uses a newer Lucene version with incompatible metadata protocols, making snapshots unreadable by OpenSearch. Use **Reindex from Remote** instead. +OpenSearch cannot restore snapshots taken by Elasticsearch 8.x, so **Reindex from Remote** is the only available method for this source version. + +The same method also works for an ES 7.10 source and, unlike Snapshot & Restore, can target OpenSearch 3.x directly without an intermediate 2.x cluster. See [Choosing a Method](#choosing-a-method) for the trade-offs. ### Prerequisites -- **Network Connectivity**: The OpenSearch cluster must be able to reach the ES 8.x cluster's HTTP/REST port (typically 9200). +- **Network Connectivity**: The OpenSearch cluster must be able to reach the source cluster's HTTP/REST port (typically 9200). +- **Index settings and mappings**: reindex copies documents only. Create the target index with the settings and mappings you need **before** reindexing, or the target index is created from dynamic mapping defaults and will not reproduce the source's shard count, custom analyzers or field types. ### Deploy ES 8.x Using ECK Operator @@ -456,13 +565,23 @@ Add the following configurations to `OpenSearchCluster` CR's `additionalConfig`: spec: general: additionalConfig: - # Allow connections to ES 8.x host (OpenSearch 3.x uses 'allowlist') + # Allow connections to the ES 8.x host. Host and port only - no http:// or https:// prefix. + # Separate multiple hosts with commas. reindex.remote.allowlist: "es8-cluster-host:9200" # Disable SSL verification for self-signed certificates reindex.ssl.verification_mode: "none" ``` -> **Note**: Nodes will be restarted after applying this configuration change. +:::warning It is `allowlist`, not `whitelist` + +Elasticsearch, and OpenSearch 1.x, used `reindex.remote.whitelist`. OpenSearch renamed the setting to `reindex.remote.allowlist` and kept the old name as a deprecated alias in 2.x. **OpenSearch 3.x removed the old name entirely**, so a configuration copied from Elasticsearch documentation makes every node fail at startup with: + +```text +SettingsException[unknown setting [reindex.remote.whitelist] ...] +``` + +Applying this change restarts the nodes one at a time. Confirm the first restarted node returns to `Running` and `Ready` before the rollout continues — see [Troubleshooting](#troubleshooting). +::: #### Step 2: Create Index Templates on OpenSearch (Optional but Recommended) @@ -517,6 +636,48 @@ curl -k -u "admin:" "https://localhost:9200/migration_test/_count" curl -k -u "elastic:" "https://es8-cluster-host:9200/migration_test/_count" ``` +## Troubleshooting + +### A node stays in CrashLoopBackOff after a configuration change + +Every entry under `spec.general.additionalConfig` is rendered as an environment variable on the pods and read by OpenSearch as a setting. The Operator does not validate these values. An unknown or misspelled setting is rejected when the node boots, and the node never starts: + +```text +[ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] uncaught exception in thread [main] +org.opensearch.bootstrap.StartupException: SettingsException[unknown setting [reindex.remote.whitelist] + please check that any required plugins are installed, or check the breaking changes documentation + for removed settings] +``` + +The Operator restarts nodes one at a time and waits for each one to become ready, so the rollout stops at the first node that fails. The remaining nodes keep running the previous configuration, which is why the cluster can still be serving traffic while one pod restarts in a loop. + +To recover: + +```bash +# 1. Identify the failing node and the rejected setting +kubectl get pods -n +kubectl logs -n -masters-0 --tail=50 + +# 2. Correct the setting in the cluster resource +kubectl edit opensearchcluster -n + +# 3. Confirm the Operator has regenerated the configuration. +# additionalConfig entries become environment variables on the pods - the Operator does +# not write an opensearch.yml ConfigMap, so check the StatefulSet, not a ConfigMap. +kubectl get sts -n - \ + -o jsonpath='{.spec.template.spec.containers[0].env}' | tr ',' '\n' | grep '' + +# 4. Restart the failing pod so it picks up the new configuration +kubectl delete pod -n -masters-0 + +# 5. Watch the rollout continue to the remaining nodes +kubectl get pods -n -w +``` + +:::note +Settings are validated one at a time, so the node reports only the first invalid setting it finds. If it fails again after your fix, repeat the procedure for the next reported setting. +::: + ## Client Migration Guide Regardless of the source ES version, **it is strongly recommended to switch to the official OpenSearch clients**.