HyperTools 1.0.0 — release to master (review & documentation; do not merge yet)

[jeremymanning]

⚠️ Do not merge yet

This PR promotes the completed HyperTools 1.0.0 line (dev-1.0) to master. It is opened for review and comprehensive change documentation; it should be merged only after maintainer sign-off, and as a squash-merge (the 1.0 development history carries large generated binaries — see Merging).

HyperTools 1.0.0 is a ground-up modernization of the toolbox. The familiar one-call API is preserved (plot, analyze, reduce, align, cluster, normalize, describe, load), but the internals, packaging, feature set, and test/security posture are new.

Diff scale (git diff master…dev-1.0): 991 files changed, +83,330 / −15,246. The hypertools/ package alone grows from a flat tools/ module set into ten focused subpackages built on one shared stack → fit-once → unstack model-application core.

This description was assembled from six parallel subsystem audits of the actual master→dev-1.0 diff, plus directly-verified demonstration figures/animations (every image below was rendered from 1.0 and visually inspected).

---

See it in action

Hyperaligned "story trajectories" — 36 subjects' whole-brain fMRI activity, hyperaligned in the 100-hub space then reduced to 3-D, tracing a shared path through a spoken story (examples/plot_story_trajectories.py, the new featured example):

Alignment — the same datasets before vs. after hyperalignment (hyp.plot(list, align='hyper')):

before	after

New-in-1.0 plotting — soft (mixture-model) clustering with blended memberships, lit convex-hull surfaces, KDE density shading, continuous multicolored lines + colorbars, the interactive plotly backend, and trajectory forecasting:

soft clustering	hull surfaces	density shading

multicolored + colorbar	interactive plotly backend	forecasting (hyp.predict)

Animated plotting (hyp.plot(list, animate=True)) and hyp.describe:

animate	describe

---

What changed, by subsystem

1. Plotting (hypertools/plot/)

Grew from 4 files to 17: low-level drawing split into two parallel renderers — matplotlib_backend.py (~2.4k LOC) and plotly_backend.py (~2.5k LOC) — sharing backend-agnostic helpers (meshutil, surface, density, morph, trails, multiindex, colors, fonts, animate) so both stay in visual lockstep. draw.py is now a 2-line shim.

New plot() keywords: backend='matplotlib'|'plotly'|'auto' (auto = plotly on Colab/Kaggle); surface= (lit, smoothed 3-D convex-hull "blob" / filled 2-D outline, Blinn-Phong lighting); density= (2-D heatmap / 3-D volumetric KDE); colorbar= (continuous or segmented-discrete); matrix/continuous hue= (multicolored lines; mixture proportions → blended RGB via mat2colors/color_reduce); names= (per-trace legend); animation styles animate='spin'|'serial'|'window'|'morph' + per-dataset lists + 2-D animation (focused=, morph_samples=); per-dataset chemtrails/precog/bullettime lists; font=/label_alpha= (CJK-aware); xlabel/ylabel/zlabel; pipeline integration (manip=, pipeline=, impute=, resample=, predict=/t=, random_state=, return_model=True → {'fig','xform_data','animation','pipeline','models','predict'}); streaming (stream_init/chunk/max/window); MultiIndex expansion; arbitrary matplotlib-prop passthrough.

Behavior: animated matplotlib plot() now returns a HyperAnimation (.to_html5_video()/.save()/notebook autoplay); defaults rotations 2→1, frame_rate 50→30; data-faithful PCHIP line interpolation; per-dataset list-kwarg length mismatches now raise rather than silently degrade. set_interactive_backend() accepts render backends.

Retired: group= (→ hue=, now raises), model=/model_params= (→ reduce={'model':…,'kwargs':…}).

2. Reduce / cluster / align / manip (hypertools/{reduce,cluster,align,manip}/)

Each rebuilt as a subpackage (main module + common.py base class); 22 new files, +6,145 LOC. Four sklearn-compatible base classes (Reducer, Clusterer, Aligner, Manipulator), each fit/transform/fit_transform/is_fitted.

• hyp.manip is new — Normalize, ZScore, Smooth (savgol|gaussian|boxcar kernels), Resample, with chaining (model=[…] builds a Pipeline).
• Cross-module stage kwargs on all four dispatchers (manip/normalize/reduce/align/cluster= + ndims=) assemble a pipeline in canonical order manip → normalize → reduce → align → cluster.
• return_model=True everywhere; a fitted model handed back is reused via .transform, never refit; random_state= injection.
• New clusterers: MeanShift, DBSCAN, OPTICS, AffinityPropagation (+ existing KMeans/…); HDBSCAN now from sklearn.cluster (no external hdbscan dep). New aligners: HyperAlign, SharedResponseModel/Det/Robust, Procrustes, NullAlign. Mixture/soft-clustering reducers (GaussianMixture/BayesianGaussianMixture/LDA/NMF) return membership proportions via the same code path as hyp.cluster. Six torch autoencoder reducers (Autoencoder, Deep/Sparse/Convolutional/Sequence/Variational).

Retired: reduce model_params; align method= and align=True ('hyper' is now a deprecated alias for 'HyperAlign'); cluster ndims= no longer silently reduces (warns without reduce=).

3. IO, data loading & dataset security (hypertools/io/)

The whole io/ package is new (load.py, sources.py, streaming.py, save.py, lsl.py), superseding master's 190-line tools/load.py.

• hyp.load resolves an 11-step chain: built-in examples → scikit-learn bundled (iris/digits/…) → seaborn → fivethirtyeight/<slug> → kaggle/<owner>/<dataset> → local files (.npy/.npz/.csv/.tsv/.txt/.json/.parquet/.mat/.xlsx/.gz) → Hugging Face ids (split=, streaming=) → Google Sheets → Google Drive → Dropbox → any URL. Failed loads emit a "tried, in order" digest with a near-miss suggestion.
• hyp.save is new — extension-aware (.csv/.npz/.parquet/.mat/.xlsx/…, else pickle), atomic (temp-file + os.replace, mode-preserving), narrowed signature (protocol= only).

Dataset security (the focus of a six-round pre-release review) — expand

• Built-ins re-hosted as non-executable formats (.npz/.parquet/.json.gz) reconstructed with no unpickling (allow_pickle=False); master fetched Google-Drive pickles and pickle.loads'd everything.
• Pinned SHA-256 verified before any deserialization; every cache hit re-validated; a mismatch is a hard error, never a silent redownload-reparse.
• Atomic, locked downloads (.part temp → hash-check → os.replace, best-effort per-dataset filelock, rate-limit-aware retry).
• Remote pickle refused unless trust=True (HypertoolsTrustError at a single chokepoint covering extension/magic-byte/protocol-0 paths); allow_pickle=False for remote npy/npz.
• sotus migrated off the datawrangler corpus-loader bypass to a plain non-executable json.gz.
• The fitted *_model sklearn pipelines are the one documented pickle exception — hash-pinned before unpickling.
• Gzip-bomb cap on transparent decompression.

Also new: streaming (io.plot_stream, issue #101) and io.lsl_stream (Lab Streaming Layer input, [lsl] extra). Vendored external/ppca.py + external/brainiak.py (SRM family) replace bare deps.

4. Forecasting, imputation & the model-application core (hypertools/{predict,impute,core}/)

Entirely new (23 files, +4,774 LOC).

• hyp.predict — Kalman, GaussianProcess, AutoRegressor, ARIMA (base install), Laplace ([predict]), Chronos ([predict-hf]); int/datetime t= horizon semantics; one model fit per dataset; return_model=True reuse (predict_new reuses learned params).
• hyp.impute — PPCA, SimpleImputer, KNNImputer, IterativeImputer, Kalman; splices (observed values byte-identical, only NaNs filled); the Kalman imputer fills fully-missing rows.
• hyp.apply_model + hyp.Pipeline — the stack → fit-once → unstack core over an explicit, eval-free model whitelist (replacing a prior eval() over sklearn's namespace); picklable, reuse-capable Pipeline unifying manip/normalize/reduce/align/cluster in canonical order.
• New HypertoolsError / HypertoolsBackendError / HypertoolsIOError; config.ini published as a queryable mirror of signature defaults.

5. Public API, packaging & CI

• __all__ added to the top-level package (star-import no longer leaks internals). New exports: manip, predict, impute, save, apply_model, supported_models, Pipeline, HyperAnimation, the io submodule, and the three exception classes. DataGeometry removed from the public surface; load/reduce/align/describe/cluster relocated to their subpackages (same public names).
• setup.py/requirements.txt/.travis.yml deleted → single pyproject.toml, Python 3.10–3.13, NumPy 2, PEP 639 SPDX license. pykalman/statsmodels moved into the base install (so predict/impute work out of the box). Optional extras: interactive, text, predict, predict-hf, io, density3d, torch, kaggle, lsl, gensim, dev. config.ini shipped as package data.
• GitHub Actions (.github/workflows/test.yml), all SHA-pinned: a 12-job matrix (Linux/Windows/macOS × py3.10–3.13) + a pandas-3 acceptance gate; wheel-smoke (installs the built wheel and sdist into fresh venvs); docs-clean (pristine git archive, sphinx -W -E -a); dataset-gate (uncached fresh download of every hosted dataset). .readthedocs.yaml updated (Python 3.11, ffmpeg + pandoc, plotly_get_chrome).

6. Examples, docs & tests

• Examples gallery: 53 scripts (+22 new) — incl. plot_datasaurus, plot_surface, plot_density, plot_colorbar, plot_multicolored_lines, plot_mixture_models, plot_predict, plot_impute, plot_autoencoders, plot_gensim_text, plot_apply_model, plot_pipelines_return_model, plot_interactive_backend, animate_plotly, animate_surface_morph, plot_story_trajectories, …
• Tutorials: the legacy .rst tutorials became executable .ipynb notebooks, +8 new (Hugging Face / Wikipedia embeddings, conversation trajectories, streaming, LSL, stock forecasting, projectile Kalman, …). Docs theme → Furo (ContextLab branding); sphinx-gallery plotly scraper + mp4 animations; api.rst reorganized into canonical pipeline order.
• Tests: ~161 files / ~2,050 tests (up from 17 files / ~136), ~15× — per-subsystem migration suites, systematic audit-fix regression suites, headless/backend + animation-export coverage, dataset integrity/compat/provenance, packaging-artifact + pickle-trust-boundary tests. Gated by a cross-platform matrix with a zero-warning -W bar.

---

Backward compatibility

The one-call API is preserved. Breaking changes are limited to long-deprecated arguments that now error instead of being silently ignored: group= → hue=; model=/model_params= → reduce=; align(method=…)/align=True → align='hyper'; cluster(ndims=…) no longer reduces. plot() returns a Figure/HyperAnimation/plotly Figure and load() returns raw data — there is no central DataGeometry object anymore (it survives only as an internal unpickle-only shim, so .geo files saved by hypertools ≥ 0.8 still load). Pre-0.8 deepdish/HDF5 geo files need a one-time out-of-process conversion (documented).

Release status

• Version is 1.0.0 (final; not .dev0). Wheel + sdist build clean, twine-valid, and both install into a fresh venv reporting hyp.__version__ == "1.0.0" (verified locally and by the CI wheel-smoke job).
• All CI green on the tip of the 1.0 line (12-job matrix + wheel/sdist smoke + uncached dataset gate + clean-docs -W).
• The branch went through six rounds of independent pre-release review (runtime security, dataset supply-chain, packaging, docs, and verification-tooling) — all findings resolved.

Merging

Please squash-merge this PR (per CONTRIBUTING.md → Merging pull requests): the 1.0 development history contains large generated binaries (rendered galleries, audit evidence, dataset payloads) that were later removed from the tree; a normal merge would copy that entire history — and every large blob in it — into master permanently. Squash-merging brings only the final tree.

Two small post-merge follow-ups (non-blocking): tag a signed v1.0.0 on the merge commit and publish only the final 1.0.0 artifacts to PyPI; the README already uses relative image paths, so nothing to re-point.

🤖 Generated with Claude Code