docs(opensearch): corrections from validating the migration guide on a live cluster - #827
Closed
SuJinpei wants to merge 3 commits into
Closed
Conversation
…on guide The guide contained several defects that break a reader who follows it verbatim: - The S3 endpoint value was written as "<http://minio.example.com:9000>" in three code samples. The angle brackets are markdown autolink syntax that leaked into the YAML value, so the S3 client points at an invalid host. Removed in all three places. - The repository-s3 download URLs hardcoded specific versions while the document itself states the plugin version must match the node version exactly. Replaced the version table with URL patterns plus the command that reads the running version, and made the warning explicit that a mismatch prevents the node from starting. - The Elasticsearch init container referenced an internal registry image that customers cannot pull. Use a public image and note that the image must match the one the nodes already run. - The Elasticsearch keystore steps read as a single-pod action. The keystore is per-node and reload_secure_settings only reloads what is already on each node, so the steps must be repeated on every pod. Also noted that the keystore is lost on restart unless config is persisted. - The snapshot request used "indices": "*" while the note below it recommended excluding system indices. The request now matches its own advice, and the note states that Kibana saved objects, users and roles therefore have to be recreated. - The reindex step extracted mappings with sed, which silently dropped the index settings (shard counts, custom analyzers). Replaced with a jq-based export of both settings and mappings. - OpenSearch was upgraded to 3.3.1 but OpenSearch Dashboards to 3.3.0. Both now use the same version. Also adds a Troubleshooting section for the case where a node stays in CrashLoopBackOff after an additionalConfig change, and a warning that reindex.remote.whitelist (the Elasticsearch name) was removed in OpenSearch 3.x in favour of reindex.remote.allowlist, which is a real failure observed on a 3.3.1 cluster. Restructures the ES 7.10 path as Phase 0 / 1 / 2 so the source-side snapshot steps are no longer an unnumbered section between the phases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… relative-link convention Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…a live cluster Ran both migration paths end to end against a real ES 7.10.2 / OpenSearch 2.19.6 / OpenSearch 3.3.1 / Elasticsearch 8.17.5 setup. The following are defects the run exposed. - The Phase 1 cluster resource omitted spec.bootstrap.pluginsList. Every general.additionalConfig entry is rendered onto the bootstrap pod as well, and the s3.client.* entries are only valid settings once repository-s3 is installed, so the bootstrap pod failed with "unknown setting [s3.client.default.region]" and the cluster never formed. The prerequisites section already showed the bootstrap block; Phase 1 did not. - The Phase 1 cluster resource supplied no admin password. OpenSearch 2.12 and later refuse to start without one, and the Operator has no dedicated field for it: security.config.adminCredentialsSecret is only the credential the Operator uses to reach the cluster, and the bootstrap pod has no env field, so the password has to be passed through additionalConfig. - The reindex verification step cited 7102099 as the version.created of an ES 7.10.2 index. The real value is 7100299. The OpenSearch-side example was also a different patch release than the text claimed, so the check is now described without a literal to compare against. - The troubleshooting recovery step said to confirm a corrected setting by grepping ConfigMaps. The Operator writes no opensearch.yml ConfigMap; additionalConfig becomes environment variables, so the check now reads the StatefulSet. Two other places describing additionalConfig as written into opensearch.yml are corrected the same way. - Reindex from Remote was presented as an ES 8.x-only method. It also works from ES 7.10 and can target OpenSearch 3.x directly, removing the need for an intermediate 2.x cluster. Added a "Choosing a Method" section covering the real trade-off (file copy vs re-indexing, and that reindex carries no settings, mappings or aliases), and listed the route in the strategy table. - The ES 8.x section attributed the snapshot incompatibility to a newer Lucene version. Measured, ES 8.17.5 and OpenSearch 2.19.6 are on the same Lucene minor, so the stated cause is unverified; the text now states the behaviour without asserting a reason. - Strengthened the Elasticsearch keystore warning: reload_secure_settings reports success for every node even when most have no credentials, and the resulting failure names the elected master rather than the pod the keystore commands were run in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SuJinpei
force-pushed
the
docs/opensearch-migration-fixes
branch
from
August 7, 2026 09:58
4f55f43 to
818144e
Compare
Contributor
Author
|
Consolidated into #826 — that PR now carries the full change set as a single commit, targeting |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #826. Stacked on that branch, so the diff here shows only the new changes.
After #826 I ran the guide end to end on a real cluster — ES 7.10.2 (3 nodes) → snapshot to S3 → OpenSearch 2.19.6 → reindex → upgrade to OpenSearch 3.3.1, plus ECK-deployed Elasticsearch 8.17.5 → reindex from remote. Everything below is a defect the run exposed, not a review comment.
The guide's core claims held up
snapshot_restore_exception: cannot restore index ... because it cannot be upgradedanalyzer [...] has not been configured in mappings; settings+mappings → createdDefects fixed here
1. Phase 1's cluster resource cannot start. Two separate omissions, each fatal:
spec.bootstrap.pluginsList. Everygeneral.additionalConfigentry is rendered onto the bootstrap pod too, ands3.client.*are only valid settings oncerepository-s3is installed, so the bootstrap pod dies withunknown setting [s3.client.default.region]. The other nodes are pinned to it throughcluster.initial_master_nodes, so the cluster never forms. The prerequisites section already showed thebootstrap:block — Phase 1 did not.No custom admin password found.security.config.adminCredentialsSecretdoes not supply it (that is only the credential the Operator uses to reach the cluster), and the bootstrap pod has noenvfield, so it has to go throughadditionalConfig.2.
version.createdfigure was wrong. The guide told readers to look for7102099on an ES 7.10.2 index; the actual value is7100299. The OpenSearch-side number was also from a different patch release than the text claimed, so the check is now written without a literal to compare against.3. The troubleshooting recovery step was wrong — it came from #826, so this corrects my own text. It said to confirm a corrected setting by grepping ConfigMaps. The Operator writes no
opensearch.ymlConfigMap;additionalConfigbecomes environment variables. The step now reads the StatefulSet, and two other places describingadditionalConfigas "written into opensearch.yml" are corrected the same way.4. Reindex from Remote was presented as ES 8.x-only. It also works from ES 7.10 and can target OpenSearch 3.x directly, which removes the entire two-hop. Verified: 5/5 documents from ES 7.10.2 straight into OpenSearch 3.3.1. Added a Choosing a Method section describing the real decision — copying index files vs re-indexing documents — including the costs of reindex (full indexing cost per document, and it carries no settings, mappings or aliases).
5. The ES 8.x Lucene explanation is unverified. The guide said ES 8.x snapshots are unreadable because of "a newer Lucene version with incompatible metadata protocols". Measured, ES 8.17.5 is on Lucene 9.12.0 and OpenSearch 2.19.6 on 9.12.3 — the same Lucene minor, so that cannot be the cause. I did not test an actual ES 8 snapshot restore, so rather than substitute a different explanation the text now states the behaviour without asserting a reason.
6. Keystore warning strengthened. Two things make the per-node requirement easy to miss:
reload_secure_settingsreports success for every node even when most have no credentials at all, and the resulting failure names the elected master, which is usually not the pod the keystore commands were run in.Notes for reviewers
docs/zhis regenerated from the English source by the pipeline.additionalConfigstores it in plain text in the cluster resource, which the warning now says explicitly, but if there is a better supported route for OpenSearch 2.12+ on Operator 2.8.x I would rather document that instead.docs/opensearch-migration-fixes(docs(opensearch): fix the ES-to-OpenSearch migration guide #826). Happy to retarget atmainonce that merges.