feat: add openhands-enterprise-troubleshooting skill for OHE self-hosted debugging - #456
Open
jpelletier1 wants to merge 28 commits into
Open
feat: add openhands-enterprise-troubleshooting skill for OHE self-hosted debugging#456jpelletier1 wants to merge 28 commits into
jpelletier1 wants to merge 28 commits into
Conversation
1 task
This was referenced Aug 26, 2026
aivong-openhands
changed the title
…bugging This skill helps diagnose and resolve common issues on OpenHands Enterprise (OHE) self-hosted installations using Replicated VM-based infrastructure. Features: - Triage and diagnosis for 8 common failure modes - Guided recovery with step-by-step resolution - Support bundle generation and parsing - Escalation handoff template for platform team Derived from Linear issue PLTF-2910.
aivong-openhands
force-pushed
the
skill/openhands-enterprise-troubleshooting
branch
from
August 31, 2026 16:16
0192d51 to
1e2b079
Compare
Contributor
Added support bundle analysisThis skill covers generating a support bundle and reading one: the real on-disk layout, how to reconstruct cluster state from it, and a script that does the mechanical first pass. What was added
Two things the tool deliberately won't do. It doesn't pretend to diagnose — it reports what the objects say and leaves to the reader which of those observations explains the user's symptom. And a clean run doesn't claim health: the guide states what a clean run actually rules out, notes that the script reads no application logs, and points to where to look next. Log analysis is deliberately left to the agent rather than the script: a fixed matcher only finds the patterns it was given and cannot tell you what it missed, whereas the guide's method — judge the format, cluster by message shape, bucket by time, read the burst — is exactly the kind of judgement an agent applies better than a regex. Nobody sends a support bundle because things are working, so "no findings" is the hard case, not the good one — and telling a broken customer their bundle looks healthy is the outcome most worth designing against. AccuracyEvery command and object name is verified against the Replicated docs and the OpenHands-Cloud charts rather than inferred, because a plausible-looking wrong command costs a support engineer more than no command at all. What that check settled:
Git provider coverage is provider-neutral. Bitbucket Data Center and Azure DevOps are first-class providers alongside GitHub and GitLab, each with its own secret and template. Both are self-hosted, so their host comes from configuration rather than a fixed domain, and an apparent auth failure is often really network or TLS — so the reader is prompted to establish which provider the user is on first. The credential check reports key names and byte lengths rather than secret values, since a present-but-empty key is the failure worth finding and Helm renders empty values into valid-looking Secrets. ValidationThe interpretation rules were checked against real Embedded Cluster bundles, including unhealthy ones (failed migrations, a crash-looping operator, Pending pods, sandbox-inflated pod counts). Healthy bundles were used only to learn what normal looks like and to catch false positives; every claim about a failure mode was checked against a bundle actually exhibiting it, or is marked unproven. Four cases where the obvious reading of the data is wrong, and which the guide therefore spells out: node conditions whose healthy value is
The triage script is a verbatim copy
The guide covers scheduling constraints as their own failure mode: a The skill ships the This comment was created by an AI agent (OpenHands) on behalf of @ai.vong. |
The support bundle section described a bundle layout that does not exist: pod-status.json, pod-logs/*.log, events.json and nodes.json are not paths troubleshoot.sh produces. The generation command was also invalid -- `replicated admin support-bundle` is a vendor-side CLI that is not present on a customer VM. Replace it with the real layout and commands: - references/support-bundle-analysis.md: the kubectl -> bundle file map, pod state reconstruction, the three independent OOM sources, the node-reboot signature, container-vs-pod log naming, redaction semantics, and the gaps that produce false negatives. - scripts/bundle_triage.py: offline first-pass triage, stdlib only. - Correct generation command for Embedded Cluster (`sudo ./openhands support-bundle`), with the pre-1.17.0 fallback. Also fixes commands in diagnostics.md that could not work as written: the Admin Console lives in the kotsadm namespace rather than replicated, and the preflight/rollback steps used vendor-side CLI calls. Rollback is now documented as the Admin Console flow it actually is, with a warning against hand-editing Helm state. Adds the .plugin manifest the skill was missing, which was failing test_all_marketplace_skills_have_plugin_json on CI, and regenerates the catalog and skills index. Co-authored-by: openhands <openhands@all-hands.dev>
aivong-openhands
force-pushed
the
skill/openhands-enterprise-troubleshooting
branch
from
August 31, 2026 16:20
1e2b079 to
c023d01
Compare
"Platform team" is internal org structure and does not name anything an external reader can act on. This skill ships publicly, so point the handoff and the bundle share at the OpenHands team instead. Also states that a support bundle is confidential. The reference already noted that the redactor over-redacts; add that over-redaction does not imply completeness -- it is pattern-based, so hostnames and user/installation identifiers routinely survive -- so a reader does not treat a heavily masked bundle as safe to paste in public. Co-authored-by: openhands <openhands@all-hands.dev>
…ndle Running the triage script against a real, healthy Embedded Cluster bundle surfaced three defects that the synthetic fixture could not, because the fixture only contained cases already anticipated. Node conditions were flagged with `type != Ready and status == True`, which assumes every non-Ready condition is negative-polarity. Vendor conditions named positively -- ContainerdHasNoDeprecations -- were therefore reported as problems on a healthy node. Only Ready and the known pressure conditions are now evaluated; unrecognised conditions are left unflagged rather than guessed at. Allocated requests summed regular and init containers together. Init containers run to completion before regular ones start, so a pod's effective request is max(sum(regular), max(init)); the additive sum overstated CPU by ~24% on the bundle tested. Native sidecars (restartPolicy: Always) keep running and are counted with the regular containers. The reference recipe disagreed with the script here and was wrong in its own way -- it omitted the phase filter, so completed Jobs inflated the total -- and is corrected too. OOMKilled containers were listed with no current state, so a kill recorded in lastState on a pod that has long since recovered read as an active incident. Entries now carry current state, readiness, and age, and only containers that are not currently healthy keep the !! marker. The three OOM sources disagree routinely: events age out of their TTL window and the analyzer keys on events, so both report clean while container state still records kills. The script printed those numbers in separate sections and left the contradiction to the reader; it now reconciles them explicitly and says which source to trust. The guide's claim that analysis.json is authoritative is qualified for the same reason. Adds log triage guidance. Most bundle logs are plain text rather than JSON, and jq aborts on the first non-JSON line, so a severity filter can print nothing for a file full of errors -- the recipes here hit exactly that and are written to avoid it. Raw counts also mislead: in one observed log 107 warnings were a single issue repeated within four seconds. Also records which parts of the guide are verified against a real bundle and which remain fixture-only, since an all-healthy bundle cannot exercise active OOM, crash loops, Pending pods, or the failing-analyzer path. Co-authored-by: openhands <openhands@all-hands.dev>
…osis A second validation pass against a fleet of real bundles, including unhealthy ones, reached paths the first healthy bundle could not. Failing analyzers were grouped by wildcarding the whole middle of the dotted name, so unrelated subsystems merged into one line: a db-cleanup failure and two warm-runtimes failures collapsed to a single heading that printed only the db-cleanup example, hiding the other two. Only identifier-shaped segments are wildcarded now. The grouping also labelled every entry FAIL regardless of severity, rendering warn-level analyzers identically to errors; entries now carry their real severity, sort worst-first, and show up to three distinct messages. The equivalent recipe in the guide over-merged the same way and is corrected. The Pending-pod projection added those pods to a baseline that already included them, since a Pending pod usually still carries a nodeName -- reporting 5.00 cores where the true total was 3.00, and biasing the EXCEEDS ALLOCATABLE flag toward false positives. Only genuinely unscheduled pods are projected now. Pending was also treated as synonymous with unschedulable. The most common real cause -- scheduled onto a node but stuck in CreateContainerConfigError or ImagePullBackOff -- has PodScheduled=True, so neither the script nor the guide reported anything for it. Both now read container and init-container waiting state as well. Validation status is updated: analyzer grouping across severities, Pending pods blocked in init, init-container attribution, and runtime-* collapse at scale are now verified against real data. Active OOM, a truly unschedulable pod, the host reboot heuristic, native sidecars, and the both-state double-count remain fixture-only, and multi-node and HA installs remain entirely unverified. Co-authored-by: openhands <openhands@all-hands.dev>
Nobody opens a support bundle because the install is working, but the tool was organised as if they might: seven sections in collection order, roughly a hundred lines, with the failure somewhere inside for the reader to spot. That is the wrong shape for something only ever run when something is broken. Output now opens with a FINDINGS block -- crash loops, active OOM kills, pods outside Running/Succeeded, unready containers, failing analyzers, bad node conditions -- ranked BROKEN NOW / DEGRADED / CONTEXT, each line naming the section that explains it. Available alone via --section findings. The ranking is mechanical and says so: it reflects what the objects report, not which finding explains the reported symptom, and a currently-OOMing container is separated from one that was killed and recovered. A clean run is now the interesting case, so it is handled explicitly rather than printing nothing. The empty state states what was actually ruled out -- pod objects, analyzer verdicts, node conditions, resource totals -- and refuses to call the install healthy, because the script reads no application logs at all. A new section covers where to look next: logs, the capture window, config rather than runtime, failed collectors, and off-cluster causes. Healthy bundles are for learning what normal looks like and catching false positives; they cannot confirm a failure mode, and the validation status now says so. Also fixes a ZeroDivisionError predating this change: a node reporting no pod capacity crashed the alloc section and took the whole run with it, which is exactly the partially-collected bundle most in need of triage. Co-authored-by: openhands <openhands@all-hands.dev>
Verified the platform-specific claims in this skill against the OpenHands-Cloud charts. Several name objects that do not exist, so the commands fail rather than mislead -- but the surrounding prose still sends the reader somewhere wrong. Podman is not supported and appears nowhere in OpenHands-Cloud. Sandboxes run under the sysbox-runc RuntimeClass, backed by a sysbox containerd runtime that a DaemonSet registers on each node. That registration failing is a real failure mode -- every sandbox stays Pending while the rest of the platform looks healthy -- and it now replaces the crictl check. The git provider section covered only GitHub and GitLab. Bitbucket Data Center is a first-class provider with its own secret and template, as is Azure DevOps. Both are self-hosted, so their host comes from configuration rather than a fixed domain, and an apparent auth failure is often really network or TLS. The section is now provider-neutral and prompts the reader to establish which provider the user is on before diagnosing. Object names are corrected to what the charts actually create. There is no git-provider-secret; each provider has its own secret (github-app, gitlab-app, bitbucket-data-center-app, azure-devops-app) consumed as environment variables by the main openhands deployment, so provider failures surface in the app's logs rather than a dedicated workload. There is no app=sandbox label or sandbox deployment either: sandboxes are created on demand by runtime-api as individual runtime-* pods. The credential check no longer prints secret values, and reports key names with byte lengths instead -- a present-but-empty key is the failure worth finding, since Helm renders an empty value into a valid-looking Secret. Co-authored-by: openhands <openhands@all-hands.dev>
The guide devotes about a hundred lines to reading logs -- format checking, clustering by message shape, time bucketing, init-container attribution -- and the script read none of them. Reading logs is most of what analysing a support bundle is, so leaving that entirely manual made the tool stop exactly where the work starts. Adds a logs section covering every canonical container log. It handles both formats in the bundle: JSON-per-line records via their severity field, and plain text via the failure patterns that carry no severity at all -- tracebacks, bare exceptions, panics, connection refusals, certificate errors. That second half matters because an uncaught exception has no severity field and often lacks the word "error", so a jq-based filter skips it silently. Results are counted by message shape rather than raw line, with volatile parts normalised, because raw counts mislead: one message repeated two hundred times is one incident. Init containers are covered, so a failed migrate-db -- which is invisible to kubectl logs <pod> and which the guide calls the easiest thing in the bundle to miss -- now surfaces without being hunted for. The canonical log tree is read directly and symlinks skipped, so the convenience trees do not double-count. The loudest logs also feed the findings block, ranked alongside broken objects rather than beneath them, since a log is the only place a failure that leaves every object healthy can appear. Coverage claims are corrected accordingly: the script no longer says it reads no logs, and now states the real limit, which is that a failure logging neither a severity field nor a recognisable pattern still will not appear. Validation status records the scan as fixture-verified only -- the patterns come from real bundles, but recall against a real log corpus is unproven. Co-authored-by: openhands <openhands@all-hands.dev>
This reverts commit 7c022ba. Log analysis is a poor fit for a fixed script and a good fit for the agent. The script can only match patterns it was told about, and its recall was unproven -- it reports what it found and nothing reports what it missed, which is the worst property for a check whose clean result invites a "no errors" conclusion. An agent reading the same logs can follow the guide's actual method: judge the format, cluster by shape, bucket by time, and read the burst in context. The guide keeps its log triage section, so the capability is documented rather than automated. Co-authored-by: openhands <openhands@all-hands.dev>
The section divides cluster-wide pod requests by the first node's allocatable, so percentages inflate with node count and EXCEEDS ALLOCATABLE stops meaning anything. No multi-node bundle is available to verify a fix against, so warn at the point of use and record the limitation rather than guess at a correction. Co-authored-by: openhands <openhands@all-hands.dev>
The public copy had diverged: a ranked findings block, defensive dict access, per-pod init-container maths and a Pending-projection fix. Fixes belong in the source skill where they can be tested against real bundles, so the script is now byte-identical and the docs describe what it does rather than what the fork did. Where the script is known to be wrong -- init containers added to the regular sum, allocatable read from the first node -- the guide says so and tells the reader how to redo the sum by hand. Node roles are documented for both live clusters and bundles, since sandboxes only schedule onto nodes labelled openhands.dev/sandbox and an unschedulable sandbox otherwise reads as a capacity problem. Co-authored-by: openhands <openhands@all-hands.dev>
The guide had grown a running commentary on how many nodes an install has. It reads as a caveat about a configuration nobody is running, and it dates the skill against a product that is still moving. Scheduling constraints are worth covering on their own terms: a Pending pod with a nodeSelector the cluster cannot satisfy is unschedulable whatever the node count, and the allocation numbers do not explain it. That guidance stays, phrased around the constraint rather than the topology. The allocation caveat keeps the init-container defect, which affects any pod with init steps, and drops the first-node-allocatable one, which cannot show up on the installs this skill is for. Co-authored-by: openhands <openhands@all-hands.dev>
…uidance Three gaps against the skill's requirements, all in prose so the triage script stays a verbatim copy of its source. The script reports observations and deliberately does not rank them, so the ranking has to live somewhere. A new subsection walks the output in the order a finding is likely to be a cause rather than a side effect, and records two ways the analyzer summary misleads: warn-severity entries print under a FAIL heading, and per-object analyzers show one arbitrary example for the whole family. Recovery said to verify at the end, which is where a half-applied fix has already buried the evidence. It now validates between steps. Log triage was written against files, but users paste excerpts. The method transfers unchanged; the hazards do not, so the two that matter are named — an excerpt is a selection made by someone with a theory, and it arrives without the container name that gives it meaning. Co-authored-by: openhands <openhands@all-hands.dev>
Contributor
Closing three gaps against the requirementsAn audit against the original requirements turned up three things the skill did not do. All three are addressed in prose — Parse and summarize the bundle to highlight the most likely root causeThe script reports observations and deliberately stops short of ranking them, so the ranking needs to live in the guide. A new Summarizing the Bundle: Most Likely Root Cause subsection walks the output in the order a finding is likely to be a cause rather than a side effect: place the symptom in time first, read Two ways the analyzer summary misleads are worth having written down, because both invite a confident wrong answer:
Two further habits the section asks for: prefer the cause nearest the symptom, since a failed Validate each recovery step before proceedingThe workflow verified once, at the end. That is the point at which a half-applied fix has already buried the evidence that would have told you the first step failed. Recovery now states what a step will change and what to expect, checks it before the next one, and stops to re-diagnose when the check disagrees. Destructive steps — restarts, rollbacks, config changes — need the user's agreement, and get recorded as they happen so the handoff can say what was touched. Raw log output pasted into the conversationLog triage was written against files, which is not how logs usually arrive. The three moves transfer to pasted text unchanged; the hazards do not, so the two that matter are named. A pasted excerpt is a selection, made by someone who already had a theory — the cause often sits in the lines just before what you were handed, so ask for the surrounding context rather than reasoning from the fragment. And it comes without a filename, so the container that produced it is unknown: the same stack trace from an init container and from the main container mean different things. The diagnostic workflow now routes all three input types — described symptom, bundle, pasted logs — from the top of Full suite: 786 passed. This comment was created by an AI agent (OpenHands) on behalf of @ai.vong. |
aivong-openhands
marked this pull request as ready for review
August 31, 2026 22:01
Contributor
|
👋 This PR needs a couple of things fixed before OpenHands can review it:
Push an update once this is addressed and this check re-runs automatically. This is an automated check - no AI was used to generate this comment. |
The Keycloak health check invoked 'kc.sh health --metrics'. No such subcommand exists — Keycloak serves health on the management port (9000) at /health/ready, and only when health is enabled, so a 404 there means the feature is off rather than the server being sick. Replaced with a port-forward and a request to the documented endpoint. The database connectivity check referenced $DB_HOST and $DB_PORT, which are defined nowhere. Inside single quotes they expanded in the pod, where they are not set either, so nc ran with empty arguments and reported a failure unrelated to the database. It now reads the pod's own configuration and then reads the error out of the log, which needs no tooling inside the container. Co-authored-by: openhands <openhands@all-hands.dev>
The health check went straight to a port-forward, which buries the question a support engineer actually has: can users reach Keycloak. The ingress answers that, and exercises DNS, TLS, and routing on the way, so a failure there localises the fault to the path rather than the server. It now leads with a request for realm metadata over the ingress. Health is the exception and is worth explaining rather than asserting. Since Keycloak 25 the /health endpoints live on a separate management interface on port 9000, which exists so that health and metrics stay off the public route; the ingress fronts the main HTTP port and does not carry them. So health means going to the pod — by exec where the image has curl, by port-forward where it does not. Separately, KEYCLOAK_URL came from a jsonpath that returns a bare host. Without a scheme curl assumes http://, so every admin-token check against a TLS-fronted Keycloak was measuring a redirect. Co-authored-by: openhands <openhands@all-hands.dev>
8 tasks
The identifiers in diagnostics.md were written from assumption rather than from a real install. Validating against five embedded-cluster bundles found ten claims that match nothing in any of them, each of which fails as a silent all-clear: a wrong selector prints "No resources found" and exits 0, so an agent reads a wrong query as a healthy component. Keycloak is in the openhands namespace as statefulset/keycloak (pod keycloak-0, selector app.kubernetes.io/name=keycloak), not a keycloak namespace with a Deployment. The app deployment is openhands; agent-server is the container image name, not a workload. LLM wiring is in openhands-litellm-config with litellm-env-secrets/openhands-env-secrets, not llm-config/llm-credentials. The TLS secret is named in the ingress spec rather than carrying an app=ingress-tls label. The replicated namespace is kURL-era and absent from embedded-cluster installs. Also stop decoding live credentials into the shell. The LLM and Keycloak sections previously assigned a working API key and admin password to shell variables, which writes them into terminal history and any agent transcript. Read key names and byte lengths instead, and run credential checks inside the pod that already holds the secret. Co-authored-by: openhands <openhands@all-hands.dev>
… live validation Validating against a live v0.58.0 / k0s 1.36.1 instance refuted several claims that five support bundles could not settle, because a bundle records object state and cannot show whether a command runs. The port-9000 health check was wrong. These installs ship Bitnami Keycloak, which does not expose the management interface upstream Keycloak 25+ serves on 9000: only 8080 and 7800 are open, :9000 refuses the connection, and :8080/health returns 404. The realm endpoint is the only reliable liveness signal, so the guidance now says to read the exposed ports and image first and treat health endpoints as distribution-specific. An earlier note here claimed 9000 was served despite not being declared in the pod spec; that rationalised away the missing containerPort instead of treating it as the evidence it was. The keycloak-admin secret claim was stated as an absolute from bundle data that showed variance. Some installs carry it mounted via KC_BOOTSTRAP_ADMIN_PASSWORD_FILE, others expose only keycloak-realm, so the guidance now says to check which is present. The app image ships no nslookup or dig, so the DNS check used getent, which is part of libc. du runs as non-root and exits 1 with permission noise even when it succeeded. kubectl is not on the PATH at all: openhands shell needs a TTY and so fails for non-interactive agents, which now have sudo k0s kubectl as the documented fallback. Co-authored-by: openhands <openhands@all-hands.dev>
…ive validation A third validation round against the live v0.58.0 instance ran the commands from the previous two commits. Most held; these did not. The LLM endpoint test was broken. LLM_API_KEY and LLM_BASE_URL are key names inside openhands-env-secrets, not variables in the app pod, which exposes LITE_LLM_API_KEY and LITE_LLM_API_URL instead. The documented request expanded to an empty host and could never have worked. The section now lists the pod environment first and explains that an empty variable in the URL means a secret key name was used in place of a pod variable name. Bitnami Keycloak wires its database through KEYCLOAK_DATABASE_* rather than upstream KC_DB_*, so the documented grep returned nothing and read as an unconfigured database. The Keycloak realm check selected the ingress by resource name. That name is correct on current builds but incidental; selecting on the auth host is stable across renames. Two error-table rows used wording that appears in no real log across six bundles: "certificate hostname mismatch" and "connection timeout" are descriptions rather than the strings OpenSSL and Go actually emit. The 120-second sandbox timeout was not observable on a real install; the dominant runtime-api timeout is a configurable 15s. Quoting a fixed number sends people after the wrong timeout, so the symptom now says to read the value. Stateful workload capture turns out to be bundle-spec-dependent: the newest bundle carries statefulsets/ while five older ones omit it, so an absent directory means not collected rather than not deployed. Co-authored-by: openhands <openhands@all-hands.dev>
Upgrades are driven from the Admin Console, and validating this section would mean deliberately breaking an upgrade on a live install, so it will not be tested. Saying so in the text is better than letting it read like the validated sections around it: an agent cannot otherwise tell that these commands carry less evidence than the ones next to them. Also stop naming kurl-proxy-kotsadm when checking console access. That is the kURL-era service name and may not exist on Embedded Cluster, the same assumption a reviewer caught for the replicated namespace. Listing the services shows what is really there. Co-authored-by: openhands <openhands@all-hands.dev>
The section keeps the operational guidance — upgrades and rollbacks are driven from the Admin Console because KOTS owns the deployment state — and drops the note about what has not been tested. Which commands happen to have been exercised is a fact about how the skill was written, not something a reader can act on. Co-authored-by: openhands <openhands@all-hands.dev>
Both upgrade warnings described the dangerous action before forbidding it — driving a rollback from the CLI, editing Helm releases, deleting resources directly. Naming a path plants it, and a reader halfway through a broken upgrade is exactly who will remember the technique rather than the prohibition. Both now say only where the work belongs: the Admin Console owns deployment state, and an install that cannot be recovered there is a support bundle and an escalation. Co-authored-by: openhands <openhands@all-hands.dev>
The condition read "if rollback is unavailable and the install is broken", which invites the reader to work out whether rollback is available before escalating. A broken install is reason enough. Co-authored-by: openhands <openhands@all-hands.dev>
Fault injection settled a question earlier rounds could not. With Keycloak scaled to zero, the documented first step returns "No resources found" and exits 0 — byte-identical to what a wrong selector returns on a healthy install. Three real selector bugs had already been found that way, so the same output has meant both things in this file. Component checks now lead with the workload READY count, which exists either way: 0/0 is scaled down, 0/1 is failing, and a NotFound error means the name was wrong. SKILL.md states the principle up front. Two of my own claims were wrong and are corrected. The kotsadm chart really does use bare app= labels, so those selectors were right all along; its app.kubernetes.io/name is admin-console, and the note now says so to stop a blind "fix" from breaking them. kurl-proxy-kotsadm is also still present on Embedded Cluster, serving 8800:30000 — the name is a kURL leftover but the service is not gone. The preflight grep failed silently for a reason unrelated to selectors: preflight logs once at deploy and had scrolled past --tail=200 on an install less than an hour old. Dropped the tail and said what silence means. Provider checks were narrow rather than broken: the bare-prefix grep misses OPENHANDS_*_SERVICE_CLS and OH_WEB_CLIENT_PROVIDERS_CONFIGURED, and the secret example assumed GitHub on installs that have GitLab. Both now follow what is actually present. Also noted that zero runtime- pods is normal on an idle install, since the doc used that count as a fault signal. Co-authored-by: openhands <openhands@all-hands.dev>
Round 5 covered the two sections no earlier round reached, and the file descriptor check was measuring three things that were not the server. ulimit -n reported the tester SSH shell (1024), fs/file-max the host ceiling, and ls /proc/self/fd the handles of the ls process (4) — while the app itself had a limit of 999999 and 17 in use. Every number looked plausible, which is why this survived five rounds. The check now reads /proc/1/limits and /proc/1/fd inside the container, and the fixes table points at the same. Certificates had the same shape of error a layer up. The public host does not resolve from the VM, so the documented s_client could not connect at all; it now targets 127.0.0.1 with the name passed as SNI. More subtly, s_client does no hostname verification by default, so a certificate for the wrong name returns "Verify return code: 0 (ok)" — the one failure the command exists to catch reads as healthy. Added -verify_hostname. The mismatch row also listed a string openssl never prints, replaced with its real lowercase wording and curl equivalent. TLS terminates in the ingress controller namespace, so a :443 service search in openhands finds nothing; the search is now cluster-wide. A blank secretName on a runtime-<id> ingress is normal rather than a fault, and now says so. App logs are structured JSON on a severity key, and Python tops out at CRITICAL, so the documented FATAL grep could never match. Left the FATAL in the Keycloak database check, where Quarkus and Postgres do emit it. kubectl debug node works but needs a TTY an agent shell may not have; on a single-node install df -h / answers the same question. Co-authored-by: openhands <openhands@all-hands.dev>
The round-5 fixes replaced commands that provably failed with commands nobody has run, and two of them fail the same silent way if wrong. The cert check now hardcoded 127.0.0.1:443, but the report that prompted it also said traefik serves 443 through a NodePort — which need not bind 443 on the host at all. Rather than swap one guess for another, find the listening port with ss first and use that. The severity grep matched "severity":"ERROR" with no space after the colon. If the log library emits a space it never matches, which is the never- matching-string defect this file has now had three times. Made the space optional and added the one-line print so the real format can be read off instead of assumed. Co-authored-by: openhands <openhands@all-hands.dev>
It was a testing diary rather than analysis guidance: three lists recording which claims had been checked against a real bundle, which were fixture-only, and which were unverified. A reader with a bundle in front of them cannot act on any of it, and it dates immediately. The one caveat in it that was operational — that --section alloc over-charges pods with init steps, so its percentages need redoing by hand — already appears where the allocation arithmetic is explained. Same for the mtime trap. Nothing actionable is lost. Co-authored-by: openhands <openhands@all-hands.dev>
…lone My previous commit replaced a working command with a dead end. It told the reader to find the TLS port with ss -lntp, but traefik publishes 443 as a NodePort, and kube-proxy in iptables mode serves those through DNAT rules rather than a listening socket — so ss matches nothing however healthy the install is, and netstat is not present as a fallback. The port comes off the ingress controller Service instead, which is where the neighbouring command was already looking. 127.0.0.1:443 was right before I changed it. -verify_hostname survives validation and stays: with the correct name it returns 0 (ok) and with a wrong one 62 (hostname mismatch), so it does detect the mismatch that plain s_client reports as healthy. It needs a bare hostname, though — a leading https:// fails the check against a good certificate, so the doc now says so and names both expected codes. The severity filter needed a caveat I would not have predicted: handled exceptions are logged at INFO with the exception name in the message, so NoCredentialsError appears in the logs while every severity field says INFO. A clean severity search therefore does not mean a quiet app. Added a text search alongside it, and extended the empty-result principle in SKILL.md to cover log filters as well as selectors. Co-authored-by: openhands <openhands@all-hands.dev>
I generalised it from five NoCredentialsError lines in one healthy install, which is most likely an optional integration going unconfigured — handled, expected, and correctly INFO. That is not evidence that error-shaped events are routinely logged at INFO, and writing it that way told readers to distrust the severity field on the strength of one narrow sample. An error the app actually hits should be logged at ERROR; if it is not, that is a bug to fix rather than a search to work around. The generic text grep further down the section already covers looking for Exception and Traceback, so nothing operational is lost. Co-authored-by: openhands <openhands@all-hands.dev>
aivong-openhands
requested review from
aivong-openhands
and removed request for
aivong-openhands
September 1, 2026 01:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Diagnosing a self-hosted OpenHands Enterprise install currently depends on knowledge that lives with individual engineers. The same eight failures recur — sandbox startup timeouts, git provider auth, certificates, LLM connectivity, Keycloak, the Replicated Admin Console, stuck upgrades, resource exhaustion — and each investigation restarts from nothing. Support bundles make this worse rather than better: they are large, and the signal is spread across
analysis.json, per-namespace pod objects, and node metrics, so reading one by hand is slow and easy to get wrong.This skill gives customers, FDEs, and first response a repeatable path from symptom to either a fix or an escalation that the platform team can act on without a second round of questions.
Summary
openhands-enterprise-troubleshootingskill: triage for the eight common failure modes, guided recovery that validates each step before the next, support bundle generation, and an escalation handoff template.references/diagnostics.md(per-failure-mode commands) andreferences/support-bundle-analysis.md(bundle layout, log triage, and how the bundle's own analyzers mislead).scripts/bundle_triage.py, which summarises a bundle offline — meta, analyzers, pods, restarts, top, alloc, events.warn-severity analyzers print under aFAILheading, and per-object analyzers show one arbitrary example per family, so[x12]is a count and not a worst case.Issue Number
Closes #527
How to Test
To exercise the script against a real bundle:
Read
SKILL.mdend to end — most of the change is prose, and its accuracy is the thing under review.Video/Screenshots
n/a — no user interface.
Notes
scripts/bundle_triage.pyis a verbatim copy of an existing internal script and is deliberately unmodified, so that improvements land at the source rather than diverging here. It has known rough edges: it raises on bundles missingnode-metricsnode info, on analyzer entries without an insight, and on nodes without pod capacity. Those are worth fixing upstream and then re-syncing.Command accuracy was checked rather than assumed: every
kubectlinvocation was parsed and run against an unreachable cluster to surface flag and subcommand errors, every single-linejqprogram was compiled, and code blocks were scanned for undefined shell variables. That found two genuine errors, both fixed here —kc.sh healthis not a Keycloak subcommand (health is HTTP on the management port, and only when enabled), and the database connectivity check referenced$DB_HOST/$DB_PORT, which are set nowhere and, inside single quotes, expanded in the pod rather than the shell.The bundle paths documented in
references/were verified against real bundles in earlier sessions; those bundles are no longer on the machine, so they were not re-verified in this pass.This description was updated by an AI agent (OpenHands) on behalf of @ai.vong.