fix(eai): don't report LLM success when the call failed

CHANGELOG.md

@@ -22,6 +22,18 @@
   `cjson` (v1.7.18), `nanopb` (v0.4.9.1), `lvgl` (v9.2.2), `tinyusb` (v0.18.0), and `unity` (v2.6.1).
 
 ### Fixed
+- **`ebuild analyze --llm` no longer reports success when the model call failed.**
+  `LLMClient` always probed `http://localhost:11434` for Ollama availability,
+  even when constructed with a different `base_url`; OpenAI-compatible URLs
+  that already ended in `/v1` were joined to `/v1/v1/chat/completions`;
+  `file://` (and other non-HTTP schemes) could reach `urllib.request.urlopen`;
+  an empty or failed response still made `ebuild analyze --llm` print
+  "LLM analysis complete". Availability now probes the configured host,
+  `/v1` is joined once, non-HTTP(S) endpoints are rejected, responses are
+  capped at 10 MB, error text redacts bearer tokens, and a failed call
+  records `llm_failed:<provider>` so the CLI warns and continues with the
+  rule-engine profile (`ebuild/eos_ai/llm_integration.py`,
+  `ebuild/eos_ai/eos_hw_analyzer.py`, `ebuild/cli/commands.py`).
 - **`ebuild test` now finds Windows test binaries.** The Ninja edge for a
   native `type: test` target already carried the platform suffix
   (`_exe_suffix()` names it `<name>.exe` on Windows), but `ebuild test`

MEMORY.md

@@ -31,7 +31,7 @@ notices the obvious-looking alternative.
 
 | Date | Decision | Reason | Rejected alternative |
 |------|----------|--------|----------------------|
-| —    | None recorded yet. | — | — |
+| 2026-09-13 | LLM endpoints accept `http` and `https`; package-index fetch stays HTTPS-only | Ollama's default listener is `http://localhost:11434`. Requiring HTTPS would break the documented local path. `file://` and other urllib schemes are rejected. | Reuse the index-sync `https://` allowlist for LLM URLs — rejected because it would disable stock Ollama. |
 
 <!-- Example of the level of detail worth recording:
 | 2026-03-14 | Queue writes in-process rather than via Redis | Deploy target has no

TASKS.md

@@ -15,6 +15,9 @@ Status is one of: `todo`, `in-progress`, `blocked`, `review`, `done`.
 | T-003 | `ebuild package` looks for the unsuffixed binary on Windows (`_build/app` rather than `_build/app.exe`) | backend | Maintenance | review | none |
 | T-004 | `_report_footprint` (the flash/RAM report `ebuild build` prints) looks for the unsuffixed binary on Windows, and fails silently rather than logging why | backend | Maintenance | review | none |
 | T-005 | Move `executable_output_path()` out of the Ninja-specific backend into a backend-neutral module (`ebuild/build/layout.py`), re-exported from `ninja_backend` for compatibility | backend | Maintenance | todo | none |
+| T-006 | Make optional LLM analysis honest: probe configured Ollama URL, join OpenAI `/v1` once, reject non-HTTP(S), do not report `--llm` success on a failed call | backend | Maintenance | review | none |
+| T-007 | Flash/RAM size regex requires `[mk]b` immediately before `flash`/`ram`, so `"2MB SPI flash"` yields `flash_size=0` and `generate_boot_yaml` silently defaults to 1 MB | backend | Maintenance | todo | none |
+| T-008 | `IndexSyncManager.sync` calls `PackageRecipe.to_dict()`, which does not exist; `tests/unit/test_index_sync.py` currently fails 9 tests on that AttributeError. Unrelated to T-006. | backend | Maintenance | todo | none |
 
 ### Evidence (self-reported by implementer; pending independent review per `.ai/reviewer.md` — "if you implemented it, you do not approve it")
 
@@ -46,9 +49,18 @@ Status is one of: `todo`, `in-progress`, `blocked`, `review`, `done`.
   process cwd's own `eos.yaml`/`board.yaml`, if any, cannot change what it
   measures; confirmed to fail against the pre-fix lookup (no report
   emitted) and pass against the fix.
-- **Suite result** (single run, both changes present, this Windows host):
-  **560 passed, 6 skipped, exit code 0**. Supersedes any other count quoted
-  for T-003 or T-004 elsewhere in this repo or in PR #110's description.
+- **T-006**: `LLMClient.is_available()` probes `self.base_url` rather than
+  hardcoded localhost; `_openai_chat_url()` joins `/v1` once; non-HTTP(S)
+  schemes never reach `urlopen`; a failed `analyze_with_llm` appends
+  `llm_failed:<provider>` and `ebuild analyze --llm` warns instead of
+  printing success. Covered by `tests/ebuild/test_llm_integration.py`
+  (**33 passed**). The `/v1` join test was confirmed to fail against the
+  pre-fix `f"{base}/v1/chat/completions"` (3 parametrized cases produced
+  `/v1/v1/chat/completions`) and pass against the fix. `llm_integration.py`
+  coverage **99.46%** on that file. Full `tests/ebuild` + `tests/unit`:
+  **699 passed, 3 skipped, 9 failed** — the 9 are pre-existing
+  `PackageRecipe.to_dict` errors in `index_sync.py`, recorded as T-008.
+
 
 ## Completed
 

docs/ai-input-formats.md

@@ -95,17 +95,23 @@ ebuild analyze notes.txt          # → Text + component DB
 # Auto-detects Ollama (local) or uses OPENAI_API_KEY from environment
 ebuild analyze "nRF52840 BLE sensor" --llm
 
-# Explicit provider
-ebuild analyze design.kicad_sch --llm-provider ollama --llm-model llama3
-ebuild analyze design.kicad_sch --llm-provider openai --llm-model gpt-4o
+# File input still needs --file; the first argument is a text description.
+ebuild analyze --file design.kicad_sch --llm
 ```
 
+`--llm-provider` and `--llm-model` are not CLI flags. Provider selection is
+environment-based (and Ollama auto-detects on localhost):
+
 **LLM Provider Auto-Detection:**
-1. **Ollama** (local, free) — checks `http://localhost:11434`
+1. **Ollama** (local, free) — probes the client's configured URL, default `http://localhost:11434`
 2. **OpenAI** — uses `OPENAI_API_KEY` env var
-3. **Custom** — uses `EOS_LLM_API_KEY` + `EOS_LLM_URL` + `EOS_LLM_MODEL` env vars
+3. **Custom** — uses `EOS_LLM_API_KEY` + `EOS_LLM_URL` + `EOS_LLM_MODEL` env vars. `EOS_LLM_URL` may be the origin (`https://api.example.com`) or the v1 root (`https://api.example.com/v1`); both resolve to `/v1/chat/completions`. Only `http://` and `https://` URLs are accepted.
 4. **None** — works without LLM (rule engine only)
 
+A failed LLM call does **not** fail the command. The rule-engine profile is
+kept, and the CLI warns that LLM analysis did not complete rather than
+printing success.
+
 ---
 
 ## NOT Supported (Today)
@@ -212,7 +218,10 @@ The generated prompt asks the LLM for:
 4. Pin assignments for detected peripherals
 5. Recommended RTOS and rationale
 
-**This is manual** — you paste the prompt into ChatGPT, Claude, or a local LLM. The LLM response is not automatically consumed by ebuild.
+`ebuild analyze --llm` sends this prompt to the configured provider and
+merges extra peripherals back into the profile. The same `llm_prompt.txt`
+is still written under `--output-dir` so you can paste it into a model
+yourself when `--llm` is off, or when the provider call fails.
 
 ---
 

docs/eos_ai_guide.md

@@ -328,3 +328,9 @@ for deeper analysis:
 Supported LLMs (all optional):
 - Local: Ollama, llama.cpp, LM Studio
 - Cloud: OpenAI, Anthropic, Grok, Gemini
+
+`ebuild analyze --llm` calls `LLMClient.auto()` (Ollama on localhost, then
+`OPENAI_API_KEY`, then `EOS_LLM_API_KEY` + `EOS_LLM_URL`). A failed call
+leaves the rule-engine profile in place and prints a warning; it does not
+report LLM analysis as complete. The generated `llm_prompt.txt` remains
+available for a manual paste.

ebuild/cli/commands.py

@@ -1844,7 +1844,15 @@ def analyze(log: Logger, input_text: Optional[str], input_file: Optional[str],
             log.info(f"  Provider: {llm_info}")
             if interpreter.llm_client.is_available():
                 profile = interpreter.analyze_with_llm(profile)
-                log.success("  LLM analysis complete")
+                if any(f.startswith("llm_analyzed:") for f in profile.features):
+                    log.success("  LLM analysis complete")
+                else:
+                    # Keep the rule-engine profile; don't print success
+                    # just because we attempted the call.
+                    log.warning(
+                        "  LLM analysis did not complete; "
+                        "continuing with the rule-engine profile"
+                    )
             else:
                 log.warning("  No LLM available. Install Ollama or set OPENAI_API_KEY.")
 

ebuild/eos_ai/eos_hw_analyzer.py

@@ -637,6 +637,11 @@ def analyze_with_llm(self, profile: HardwareProfile) -> HardwareProfile:
 
         This is optional — the analyzer works without any LLM.
         Returns the original profile unchanged if no LLM is available.
+
+        On a successful call, appends ``llm_analyzed:<provider>`` to
+        ``profile.features``. On a failed call, appends
+        ``llm_failed:<provider>`` and leaves peripherals unchanged, so a
+        caller can tell the two cases apart.
         """
         if not self.llm_client.is_available():
             return profile
@@ -645,6 +650,8 @@ def analyze_with_llm(self, profile: HardwareProfile) -> HardwareProfile:
         response = self.llm_client.analyze(prompt)
 
         if not response.success:
+            # Distinguish "call failed" from "LLM was never asked".
+            profile.features.append(f"llm_failed:{self.llm_client.provider}")
             return profile
 
         # Parse LLM response for additional peripheral recommendations

ebuild/eos_ai/llm_integration.py

@@ -27,10 +27,85 @@
 
 import json
 import os
+import re
 import urllib.error
 import urllib.request
 from dataclasses import dataclass
 from typing import Optional
+from urllib.parse import urlparse
+
+# Same ceiling the package-index fetcher uses. An LLM analysis is a YAML
+# recommendation, not a stream; a response larger than this is a fault, not
+# a useful answer, and urllib would otherwise read it into memory unbounded.
+MAX_LLM_RESPONSE_BYTES = 10 * 1024 * 1024
+
+_ALLOWED_SCHEMES = {"http", "https"}
+_BEARER_RE = re.compile(r"Bearer \S+", re.IGNORECASE)
+_KEY_QUERY_RE = re.compile(
+    r"(api[_-]?key|token|secret)=([^&\s]+)", re.IGNORECASE
+)
+
+
+def _http_url(url: str) -> str:
+    """Return *url* if it names an HTTP(S) endpoint.
+
+    ``urllib.request.urlopen`` opens ``file://`` and other schemes. An LLM
+    endpoint is an HTTP service; anything else is either a programming error
+    or SSRF. Unlike the package index, HTTP (not only HTTPS) is allowed:
+    Ollama's default listener is ``http://localhost:11434``.
+    """
+    parsed = urlparse(url)
+    if parsed.scheme.lower() not in _ALLOWED_SCHEMES or not parsed.netloc:
+        scheme = parsed.scheme or "empty"
+        raise ValueError(f"LLM endpoint must be an http(s) URL (got {scheme})")
+    return url
+
+
+def _openai_chat_url(base_url: str) -> str:
+    """Join *base_url* to the chat-completions path without doubling ``/v1``.
+
+    OpenAI-compatible servers document the base as either the origin
+    (``https://api.openai.com``) or the v1 root (``https://api.openai.com/v1``).
+    Always appending ``/v1/chat/completions`` made the second form 404.
+    """
+    base = _http_url(base_url).rstrip("/")
+    if base.endswith("/v1"):
+        return f"{base}/chat/completions"
+    return f"{base}/v1/chat/completions"
+
+
+def _error_text(exc: BaseException) -> str:
+    """Render *exc* for the caller without echoing credentials."""
+    text = str(exc)
+    text = _BEARER_RE.sub("Bearer [redacted]", text)
+    text = _KEY_QUERY_RE.sub(r"\1=[redacted]", text)
+    return text
+
+
+def _read_limited(resp, limit: int = MAX_LLM_RESPONSE_BYTES) -> bytes:
+    data = resp.read(limit + 1)
+    if len(data) > limit:
+        raise ValueError(f"LLM response exceeded {limit} bytes")
+    return data
+
+
+def _completed(text: str, model: str, provider: str, tokens_used: int) -> LLMResponse:
+    if not text.strip():
+        return LLMResponse(
+            text="",
+            model=model,
+            provider=provider,
+            tokens_used=tokens_used,
+            success=False,
+            error="LLM returned an empty response",
+        )
+    return LLMResponse(
+        text=text,
+        model=model,
+        provider=provider,
+        tokens_used=tokens_used,
+        success=True,
+    )
 
 
 @dataclass
@@ -111,26 +186,41 @@ def auto(cls) -> "LLMClient":
         return cls(provider="none", model="none")
 
     @staticmethod
-    def _check_ollama() -> bool:
-        """Check if Ollama is running locally."""
+    def _check_ollama(base_url: Optional[str] = None) -> bool:
+        """Return True if an Ollama server answers at *base_url*.
+
+        Availability must probe the URL the client will actually call. The
+        previous probe always hit ``OLLAMA_URL`` (localhost:11434), so a
+        client constructed with ``base_url=http://gpu-box:11434`` reported
+        itself available whenever a *different* Ollama was running locally,
+        and unavailable when only the configured host was up.
+        """
+        url = (base_url or LLMClient.OLLAMA_URL).rstrip("/")
         try:
-            req = urllib.request.Request(
-                f"{LLMClient.OLLAMA_URL}/api/tags",
-                method="GET",
-            )
+            _http_url(url)
+        except ValueError:
+            return False
+        try:
+            req = urllib.request.Request(f"{url}/api/tags", method="GET")
             with urllib.request.urlopen(req, timeout=3) as resp:
                 return resp.status == 200
-        except (urllib.error.URLError, OSError, TimeoutError):
+        except (urllib.error.URLError, OSError, TimeoutError, ValueError):
             return False
 
     def is_available(self) -> bool:
         """Check if this LLM client can make requests."""
         if self.provider == "none":
             return False
         if self.provider == "ollama":
-            return self._check_ollama()
+            return self._check_ollama(self.base_url)
         if self.provider in ("openai", "custom"):
-            return bool(self.api_key and self.base_url)
+            if not (self.api_key and self.base_url):
+                return False
+            try:
+                _http_url(self.base_url)
+            except ValueError:
+                return False
+            return True
         return False
 
     def analyze(self, prompt: str, system: str = "") -> LLMResponse:
@@ -161,17 +251,17 @@ def analyze(self, prompt: str, system: str = "") -> LLMResponse:
         try:
             if self.provider == "ollama":
                 return self._call_ollama(prompt, system)
-            else:
-                return self._call_openai_compat(prompt, system)
+            return self._call_openai_compat(prompt, system)
         except Exception as e:
             return LLMResponse(
                 text="", model=self.model, provider=self.provider,
-                success=False, error=str(e),
+                success=False, error=_error_text(e),
             )
 
     def _call_ollama(self, prompt: str, system: str) -> LLMResponse:
         """Call Ollama local API."""
-        url = f"{self.base_url}/api/generate"
+        base = _http_url(self.base_url or "").rstrip("/")
+        url = f"{base}/api/generate"
         payload = {
             "model": self.model,
             "prompt": prompt,
@@ -187,19 +277,18 @@ def _call_ollama(self, prompt: str, system: str) -> LLMResponse:
         )
 
         with urllib.request.urlopen(req, timeout=self.timeout) as resp:
-            body = json.loads(resp.read().decode("utf-8"))
+            body = json.loads(_read_limited(resp).decode("utf-8"))
 
-        return LLMResponse(
-            text=body.get("response", ""),
+        return _completed(
+            text=body.get("response") or "",
             model=body.get("model", self.model),
             provider="ollama",
-            tokens_used=body.get("eval_count", 0),
-            success=True,
+            tokens_used=int(body.get("eval_count") or 0),
         )
 
     def _call_openai_compat(self, prompt: str, system: str) -> LLMResponse:
         """Call OpenAI-compatible chat completions API."""
-        url = f"{self.base_url}/v1/chat/completions"
+        url = _openai_chat_url(self.base_url or "")
         payload = {
             "model": self.model,
             "messages": [
@@ -218,18 +307,18 @@ def _call_openai_compat(self, prompt: str, system: str) -> LLMResponse:
         req = urllib.request.Request(url, data=data, headers=headers, method="POST")
 
         with urllib.request.urlopen(req, timeout=self.timeout) as resp:
-            body = json.loads(resp.read().decode("utf-8"))
+            body = json.loads(_read_limited(resp).decode("utf-8"))
 
-        choice = body.get("choices", [{}])[0]
-        message = choice.get("message", {})
-        usage = body.get("usage", {})
+        choices = body.get("choices") or [{}]
+        choice = choices[0] if choices else {}
+        message = choice.get("message") or {}
+        usage = body.get("usage") or {}
 
-        return LLMResponse(
-            text=message.get("content", ""),
+        return _completed(
+            text=message.get("content") or "",
             model=body.get("model", self.model),
             provider=self.provider,
-            tokens_used=usage.get("total_tokens", 0),
-            success=True,
+            tokens_used=int(usage.get("total_tokens") or 0),
         )
 
     def get_provider_info(self) -> str: