HyperTools 1.0.0 — release to master (review & documentation; do not merge yet)

.gitignore

@@ -14,3 +14,22 @@ tox.ini
 \.idea/
 docs/_build
 .pytest_cache/
+.venv/
+venv/
+.omc/
+__pycache__/
+tests/screenshots/
+docs/auto_examples/animation.mp4
+docs/auto_examples/searchindex.db
+docs/auto_examples/test-image.pdf
+
+# stray local virtualenvs / build outputs (audit X5-001/X5-006)
+hypertools-dev/
+dist/
+
+# audit working notes + evidence (moved to PR #272 comments; full trail on
+# the audit/release-1.0-2026-07 branch). Do not re-track (release review #10).
+notes/audit-1.0-2026-07/
+# generated sphinx-gallery output: regenerated at docs build time from
+# examples/*.py -- source examples stay tracked, generated gallery does not.
+docs/auto_examples/

.readthedocs.yaml

@@ -9,9 +9,28 @@ version: 2
 build:
   os: ubuntu-22.04
   tools:
-    python: "3.9"
+    python: "3.11"
     # You can also specify other tools here or in a separate tools block
+  # ffmpeg system binary: required by sphinx-gallery's
+  # matplotlib_animations = (True, 'mp4') setting in docs/conf.py to render
+  # the animated gallery examples as mp4 video
+  apt_packages:
+    - ffmpeg
+    # pandoc: nbsphinx renders the hand-authored tutorials/*.ipynb via pandoc.
+    # RTD's build image has historically bundled pandoc, but declare it
+    # explicitly so the notebook stage never depends on an implicit default
+    # (the docs-clean CI job installs it the same way).
+    - pandoc
   jobs:
+    post_install:
+      # Kaleido 1.x renders the gallery's Plotly examples (plot_interactive_backend,
+      # animate_plotly, ...) to static PNGs via HEADLESS CHROME -- the
+      # plotly_sg_scraper wired up in docs/conf.py. Read the Docs provisions no
+      # browser, so without this the two Plotly examples fail when Kaleido tries
+      # to launch Chrome and Sphinx exits non-zero (2026-07 release review,
+      # blocker #2). Fetch a pinned Chrome into choreographer's deps dir, where
+      # Kaleido discovers it automatically (no PATH wiring needed).
+      - plotly_get_chrome -y
     post_build:
       # Fix animated GIF thumbnails after documentation build
       - python docs/post_build.py

CHANGELOG.md

@@ -0,0 +1,132 @@
+# Changelog
+
+## 1.0.0 (2026-07-18)
+
+HyperTools 1.0 is a ground-up modernization of the toolbox. The familiar
+one-call API (`plot`, `analyze`, `reduce`, `align`, `cluster`, `normalize`,
+`describe`, `load`) is preserved, but the internals, packaging, and feature
+set are new.
+
+### Architecture
+
+- **Package restructure:** the old monolithic `tools` module is split into
+  focused subpackages -- `reduce/`, `cluster/`, `align/`, `manip/`, `io/`,
+  `predict/`, `impute/`, `plot/`, and `core/` -- all built on a shared
+  stack -> fit-once -> unstack model-application core (`hyp.apply_model`,
+  backed by [pydata-wrangler](https://github.com/ContextLab/data-wrangler)).
+- **Canonical pipeline order:** every dispatcher composes stages in the
+  same order (manip -> normalize -> reduce -> align -> cluster), and
+  `return_model=True` returns fitted models/`hyp.Pipeline` objects that can
+  be replayed on new data via `pipeline=`.
+- **Return types:** `hyp.plot` returns a matplotlib `Figure` (a
+  `HyperAnimation` when `animate=` is used, or a plotly `Figure` with
+  `backend='plotly'`); `hyp.load` returns raw data. The 0.x `DataGeometry`
+  ("geo") container is retired to an internal unpickle-only shell so
+  **pickle-format geo files saved by hypertools ≥0.8 still load** (returning
+  their raw data). Older **pre-0.8 `deepdish`/HDF5-format** geo files cannot
+  be read under the required NumPy 2 and must be converted once in a
+  throwaway `numpy<2` environment (`hyp.load` detects them and explains how;
+  see the README "Legacy data" note).
+- **Packaging:** `pyproject.toml`-based packaging, Python 3.10-3.13, and a
+  base install that covers all core functionality and therefore pulls in
+  the full scientific stack (NumPy, SciPy, pandas, scikit-learn,
+  matplotlib, seaborn, UMAP/Numba, statsmodels, pykalman, pydata-wrangler)
+  -- not a minimal footprint, but fast-importing (heavy dependencies load
+  lazily, so `import hypertools` is roughly 3.5x faster than 0.8.x). Heavier
+  optional model families are separated into extras (`interactive`, `text`,
+  `predict`, `predict-hf`, `io`, `density3d`, `torch`, `kaggle`, `lsl`,
+  `gensim`, `dev`).
+
+### New features
+
+- Interactive plotly backend (`backend='plotly'`; `backend='auto'` selects
+  plotly automatically on Google Colab/Kaggle), visually matched to the
+  matplotlib backend.
+- New animation styles (`'spin'`, `'serial'`, `'window'`, `'morph'`), 2-D
+  animation support, and per-dataset `chemtrails`/`precog`/`bullettime`
+  trails.
+- Hull surfaces (`surface=`), density shading (`density=`), colorbars,
+  multicolored lines via continuous/matrix `hue`, nested-list input, and
+  automatic MultiIndex DataFrame expansion.
+- Mixture-model ("soft") clustering (`GaussianMixture`,
+  `BayesianGaussianMixture`, `LatentDirichletAllocation`, `NMF`).
+- `hyp.manip` (Normalize/ZScore/Smooth/Resample) with chaining,
+  `hyp.predict` timeseries forecasting (Kalman, ARIMA, GP, autoregression,
+  Laplace, Chronos), and `hyp.impute` missing-data imputation (PPCA,
+  Kalman, sklearn imputers).
+- Optional torch-backed autoencoder reducers, gensim text
+  vectorizers/semantic models, Lab Streaming Layer input
+  (`hyp.io.lsl_stream`), and new `hyp.load` sources (Kaggle, Hugging Face,
+  Google Sheets/Drive, Dropbox, URLs, and more local formats).
+- Reproducibility via a top-level `random_state=` on
+  `reduce`/`cluster`/`analyze`/`plot`.
+
+### Removed / changed behavior
+
+- Retired legacy arguments now raise errors instead of being silently
+  accepted: `group=` (use `hue=`), `model=`/`model_params=` (use
+  `reduce=`), and `align=True`/`align(method=...)` (use `align='hyper'`,
+  `align='SRM'`, etc.). `cluster`'s `ndims=` is only a passthrough to
+  `reduce=` and warns if passed without it.
+- Plotting no longer mutates global matplotlib settings; the unreliable
+  result cache was removed; HDBSCAN comes from scikit-learn instead of the
+  external `hdbscan` package.
+- **Typography:** plots now render in a bundled sans-serif (Noto Sans, SIL
+  OFL 1.1, vendored in `hypertools/external/fonts`). The **matplotlib**
+  backend is handed the font FILE, so it renders in Noto Sans identically on
+  every platform instead of inheriting the machine's default face. The
+  **plotly** backend can only pass a family NAME to the rendering browser
+  (never a font file), so it *prefers* Noto Sans but falls back to the next
+  installed system face when Noto isn't present -- plotly typography can
+  still vary by platform. Fonts resolve through a per-glyph FALLBACK STACK,
+  so text mixing scripts renders completely from several faces rather than
+  showing "tofu" boxes for whatever the primary face lacks, and the primary
+  face stays Noto Sans -- an accent or Greek letter no longer swaps the whole
+  plot onto some other installed font. A covering font is auto-added to the
+  stack (as a fallback, Noto still primary) only when the stack genuinely
+  cannot draw a character. Also: point `labels=` no longer force a serif
+  face (they inherit the stack like every other text surface -- previously a
+  label character the serif faces lacked rendered as tofu even when an
+  installed font had it), and the "no font covers this text" warning now
+  fires only for characters NOTHING available can draw, instead of whenever
+  no SINGLE font covered all of it. The font stack is applied inside a scoped
+  `rc_context`, so your own matplotlib settings are left untouched.
+- **Animation controls (plotly):** the Play/Pause buttons moved from the
+  plot's bottom-left corner to below the plotting area, laid out
+  horizontally and lightly themed. In 2-D -- where the axes fill the paper
+  area -- they previously overlapped the chart itself.
+- **Frame outline weight (plotly):** the 3-D wireframe cube and the 2-D
+  square frame now render at the same ~2px stroke (matching the matplotlib
+  backend). plotly's gl line renderer draws 3-D `Scatter3d` lines lighter
+  than the equivalent 2-D SVG shape, so the 3-D cube previously looked
+  noticeably thinner than the 2-D square.
+
+### Release audit (2026-07)
+
+Before release, the codebase, documentation, examples, and tutorials were
+red-teamed in a 46-unit audit that filed 708 findings (691 confirmed by an
+independent verifier). The confirmed code findings were fixed in waves
+(350+ fixes merged as of this entry), including these criticals:
+
+- `hyp.load('sotus')` returns the full 29-speech State of the Union corpus
+  again (the hosted corpus had been loading incompletely).
+- `hyp.align` preserves each dataset's row order (aligned outputs are no
+  longer returned with scrambled rows).
+- `hyp.manip` smoothing runs per dataset: `Smooth` kernels no longer bleed
+  across dataset boundaries when given a list.
+- The Kalman forecaster (`hyp.predict(..., model='Kalman')`) actually
+  learns its dynamics model instead of filtering with default parameters.
+- CSV/TSV parsing bugs in `hyp.load` were fixed, and `hyp.save` writes
+  atomically and format-aware.
+- `import hypertools` no longer crashes under unusual
+  backend-related environment variable configurations.
+- Plotting nested lists of datasets (`hyp.plot([[a, b], [c]])`) works
+  correctly again.
+
+Docs, README, examples, and tutorials were then re-verified by executing
+them against the fixed code.
+
+## 0.8.x and earlier
+
+See [RELEASE_NOTES_0.8.1.md](RELEASE_NOTES_0.8.1.md) and the
+[GitHub releases page](https://github.com/ContextLab/hypertools/releases).

CLAUDE.md

@@ -4,18 +4,17 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Project Overview
 
-HyperTools is a Python library for visualizing and manipulating high-dimensional data. It provides a unified interface for dimensionality reduction, data alignment, clustering, and visualization, built on top of matplotlib, scikit-learn, and seaborn.
+HyperTools is a Python library for visualizing and manipulating high-dimensional data. It provides a unified interface for dimensionality reduction, data alignment, clustering, and visualization, built on top of matplotlib, plotly (interactive backend), scikit-learn, seaborn, and pydata-wrangler (core data-wrangling).
 
 ## Key Commands
 
 ### Testing
-- `pytest` - Run all tests from the hypertools/ directory
+- `pytest` - Run all tests (run from the repo root; `pyproject.toml` sets `testpaths = ["tests"]`)
 - `pytest tests/test_<module>.py` - Run tests for a specific module
 - `pytest tests/test_<module>.py::test_<function>` - Run a specific test function
 
 ### Development Setup
-- `pip install -e .` - Install in development mode
-- `pip install -r requirements.txt` - Install dependencies
+- `pip install -e ".[dev]"` - Install in development mode with test/dev dependencies. `[dev]` does NOT pull in every optional extra: `text` (pydata-wrangler[hf], HF text embeddings) and `predict-hf` (chronos-forecasting) are left out, and their code paths are `importorskip`-guarded in tests.
 - `pip install -r docs/doc_requirements.txt` - Install documentation dependencies
 
 ### Documentation
@@ -27,50 +26,78 @@ HyperTools is a Python library for visualizing and manipulating high-dimensional
 ### Core Components
 
 **DataGeometry Class** (`hypertools/datageometry.py`)
-- Central data container that holds raw data, transformed data, and transformation parameters
-- Stores matplotlib figure/axes handles and animation objects
-- Contains normalization, reduction, and alignment model parameters
+- INTERNAL, unpickle-only legacy shell (its own docstring: "not part of the public API"); not used by any current API function
+- Kept solely so `hypertools.load()` can unpickle hosted example-dataset geo files (created by hypertools < 1.0) and extract their raw data via `get_data()`
+- In 1.0, `plot()` returns a matplotlib `Figure` and `load()` returns raw data -- no `DataGeometry` is ever constructed or returned to users
 
 **Main API Functions** (`hypertools/__init__.py`)
 - `plot()` - Primary visualization function
-- `analyze()` - Data analysis and dimensionality reduction
+- `analyze()` - Classic manip -> normalize -> reduce -> align -> cluster pipeline dispatcher
 - `reduce()` - Dimensionality reduction utilities
 - `align()` - Data alignment across datasets
 - `normalize()` - Data normalization
 - `describe()` - Data description and summary
 - `cluster()` - Clustering functionality
+- `manip()` - Manipulator dispatcher (Normalize/ZScore/Smooth/Resample, etc.)
+- `predict()` - Timeseries forecasting (ARIMA, Kalman, GP, autoregression, Chronos, etc.)
+- `impute()` - Missing-data imputation (PPCA, Kalman, sklearn imputers)
 - `load()` - Data loading utilities
+- `save()` - Save data/results to disk
+- `apply_model()` - Apply an arbitrary sklearn-API model to data
+- `Pipeline` - Class for chaining/reusing manip/reduce/align/cluster stages
+- `set_interactive_backend()` - Switch the plotting backend (matplotlib/plotly)
+- `io` submodule - I/O helpers, including `io.lsl_stream()` for Lab Streaming Layer input
 
 **Tools Module** (`hypertools/tools/`)
-- `align.py` - Hyperalignment and Procrustes alignment
-- `reduce.py` - Dimensionality reduction (PCA, t-SNE, UMAP, etc.)
-- `normalize.py` - Data normalization methods
-- `cluster.py` - K-means and other clustering algorithms
+- `align.py` - Classic array/mode `align()` API; thin compatibility wrapper over `hypertools/align/`
+- `analyze.py` - `analyze()` pipeline dispatcher
+- `normalize.py` - `normalize()` (z-scoring) with cross-module stage kwargs
 - `format_data.py` - Data preprocessing and formatting
 - `text2mat.py` - Text-to-matrix conversion
 - `df2mat.py` - DataFrame-to-matrix conversion
-- `load.py` - Data loading from various sources
+- `gensim_models.py` - sklearn-API wrappers around gensim topic/embedding models
 - `missing_inds.py` - Missing data handling
-- `procrustes.py` - Procrustes analysis
+
+The dev-1.0 refactor moved several tools into their own top-level subpackages (each a folder with a main module plus helpers):
+- `hypertools/reduce/` - Dimensionality reduction (`reduce.py`, `describe.py`, `autoencoders.py`, `common.py`)
+- `hypertools/cluster/` - Clustering (`cluster.py`, `common.py`)
+- `hypertools/align/` - Alignment (`align.py`, `hyperalign.py`, `procrustes.py`, `srm.py`, `null.py`, `common.py`)
+- `hypertools/manip/` - Manipulators (`manip.py`, `normalize.py`, `zscore.py`, `smooth.py`, `resample.py`, `common.py`)
+- `hypertools/io/` - Loading/saving/streaming (`load.py`, `save.py`, `sources.py`, `streaming.py`, `lsl.py`)
+- `hypertools/predict/` - Forecasting models (`predict.py`, `arima.py`, `autoreg.py`, `gp.py`, `kalman.py`, `laplace.py`, `chronos.py`, `common.py`)
+- `hypertools/impute/` - Imputation models (`impute.py`, `ppca.py`, `kalman.py`, `sklearn_imputers.py`, `common.py`)
+- `hypertools/core/` - Shared config/exceptions and `apply_model()`/`Pipeline` (`configurator.py`, `exceptions.py`, `model.py`, `pipeline.py`, `shared.py`)
 
 **Plot Module** (`hypertools/plot/`)
 - `plot.py` - Main plotting interface and logic
-- `backend.py` - matplotlib backend configuration
-- `draw.py` - Low-level drawing functions
-
-**External Dependencies** (`hypertools/_externals/`)
-- `ppca.py` - Probabilistic Principal Component Analysis
-- `srm.py` - Shared Response Model
+- `backend.py` - Backend selection (`set_interactive_backend()`)
+- `matplotlib_backend.py` - Low-level matplotlib drawing (`draw.py` is now a 3-line compatibility shim over this)
+- `plotly_backend.py` - Low-level plotly drawing (interactive backend)
+- `interactive.py` - 4-line rename shim over `plotly_backend.py` (preserves the pre-1.0 `hypertools.plot.interactive` import path)
+- `animate.py` - Animation support
+- `colors.py` - Color handling
+- `density.py` / `surface.py` - Density and iso-surface rendering
+- `morph.py` / `trails.py` - Morph and trail effects
+- `meshutil.py` - Mesh utilities
+- `multiindex.py` - Pandas MultiIndex handling
+- `fonts.py` - Font handling
+
+**Vendored Third-Party Code** (`hypertools/external/`)
+- `ppca.py` - Probabilistic Principal Component Analysis (vendored from pca-magic, Apache-2.0)
+- `brainiak.py` - Shared Response Model family (vendored from brainiak, Apache-2.0)
+- `hypertools/_externals/` contains only 3-line re-export shims (`ppca.py`, `srm.py`) that preserve the pre-1.0 import paths; the real implementations live in `hypertools/external/`
 
 ### Data Flow
 
 1. **Input Processing**: Data is formatted and validated through `format_data()`
 2. **Normalization**: Optional data normalization via `normalize()`
-3. **Alignment**: Optional cross-dataset alignment via `align()`
-4. **Dimensionality Reduction**: Data is reduced via `reduce()`
+3. **Dimensionality Reduction**: Data is reduced via `reduce()`
+4. **Alignment**: Optional cross-dataset alignment via `align()`
 5. **Clustering**: Optional clustering via `cluster()`
 6. **Visualization**: Final plotting through `plot()`
 
+(Verified canonical order — audit F03-011: normalize → reduce → align → cluster; figure coordinates match this staged order exactly.)
+
 ### Key Design Patterns
 
 - **Modular Architecture**: Each major operation (align, reduce, normalize, etc.) is in its own module
@@ -83,10 +110,10 @@ HyperTools is a Python library for visualizing and manipulating high-dimensional
 
 - The package follows a functional programming style with separate modules for each operation
 - All major functions are designed to work with multiple input formats
-- The DataGeometry class serves as the central data container and state manager
+- `plot()` returns a matplotlib `Figure`; `load()` returns raw data -- there is no central state-container object in 1.0 (`DataGeometry` is an internal unpickle-only legacy shell; see above)
 - Tests are located in `tests/` directory and follow pytest conventions
 - Documentation is built with Sphinx and uses example galleries
-- The codebase maintains compatibility with Python 3.9+
+- The codebase requires Python 3.10+ (`requires-python = ">=3.10"`)
 
 ## Testing Strategy