People on stage and camera motion, read off the video

CHANGELOG.md

@@ -15,6 +15,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   `build_pyramid` first. It now builds the levels on first use.
 
 ### Added
+- **People on stage** — new `_performers` module. `detect_people` runs a YOLO person detector
+  (`ultralytics` extra) at a low frame rate through an ffmpeg pipe; `on_stage` keeps the boxes
+  whose heads are in the upper part of the frame and drops the audience cut off by the bottom
+  edge; `performer_count` reports how many perform in a span as a high percentile of the
+  per-frame counts (the wide shots), with median and maximum beside it. Exact for a soloist,
+  a duo and a five-piece band from an operated concert camera; a choir is under-counted.
+- **Camera cuts and PTZ** — new `_camera` module. `camera_motion` labels each sample of a video
+  `still`, `moving` or `cut` from ORB matches and a partial-affine RANSAC fit between consecutive
+  frames of a small 2 fps proxy (`make_proxy`), and returns cuts, shots and the share of time in
+  each state; `still_runs` gives the *framings* (still runs between moves and cuts);
+  `camera_state_at` samples the state at arbitrary times, so motion measures can exclude camera
+  motion. `performer_count(..., camera=...)` counts per framing (the 75th percentile of each, the
+  widest framing being the estimate), which is what makes the count right with an operated camera.
 - `tracks.json` (and the dict both extractors return) carries `analysis_dir`, so a caller can
   go from `extract_tracks_parallel(...)` to `read_columns`/`check_tracks` without
   reconstructing the `analysis/<stem>` path convention.

docs/MODULES.md

@@ -16,6 +16,7 @@ Each page is rendered from the source docstrings by
 - [Audiofeatures](musicalgestures/_audiofeatures.md)
 - [Blend](musicalgestures/_blend.md)
 - [Blurfaces](musicalgestures/_blurfaces.md)
+- [Camera](musicalgestures/_camera.md)
 - [CenterFace](musicalgestures/_centerface.md)
 - [Cli](musicalgestures/cli.md)
 - [Co-accentuation](musicalgestures/_coaccentuation.md)
@@ -59,6 +60,7 @@ Each page is rendered from the source docstrings by
 - [Movementbeats](musicalgestures/_movementbeats.md)
 - [Package overview](musicalgestures/index.md)
 - [Peaks](musicalgestures/_peaks.md)
+- [Performers](musicalgestures/_performers.md)
 - [Physio](musicalgestures/_physio.md)
 - [Pipeline](musicalgestures/_pipeline.md)
 - [Pose timeline](musicalgestures/_posetimeline.md)

docs/musicalgestures/_camera.md

@@ -0,0 +1,3 @@
+# Camera cuts and PTZ
+
+::: musicalgestures._camera

docs/musicalgestures/_performers.md

@@ -0,0 +1,3 @@
+# People on stage
+
+::: musicalgestures._performers

docs/user-guide/people-and-camera.md

@@ -0,0 +1,49 @@
+# People on stage and camera motion
+
+A concert video answers two questions a motion measure cannot: *how many people are
+performing*, and *is the picture moving because they move or because the camera does*.
+Both are read from the video alone.
+
+## Counting who is on stage
+
+```python
+import musicalgestures as mg
+
+det = mg.detect_people("concert.mp4", fps=1.0)          # YOLO person boxes, once per second
+mg.performer_count(det, start_s=378, end_s=668)          # {'estimate': 1, 'median': 1, 'max': 1, ...}
+```
+
+A person detector finds the audience too: heads and shoulders in the lower part of the
+frame, cut off by the bottom edge. Performers stand or sit on a raised stage, so their
+heads are in the upper half; `on_stage` keeps those boxes and drops the rest. Because an
+operated camera rarely shows everyone at once, the span statistic is a high percentile of
+the per-frame counts (the wide shots), reported with the median and maximum so the
+variation in framing stays visible.
+
+## Camera cuts and pan/tilt/zoom
+
+```python
+cam = mg.camera_motion("concert.mp4")                    # on a 2 fps proxy, about a minute per hour
+cam["summary"]                                           # {'still': 0.86, 'moving': 0.14, 'cut': 0.002}
+cam["cuts"], cam["shots"]                                # cut times, spans between cuts
+mg.still_runs(cam, 378, 668)                             # the framings: still runs between moves and cuts
+mg.camera_state_at(cam, [400.0, 401.0])                  # 'still' | 'moving' | 'cut'
+```
+
+Between consecutive frames of the proxy, ORB features and a partial-affine RANSAC fit give
+a translation, a scale change and an inlier count. Consistent geometry with a shift or a
+zoom is a camera move; no consistent geometry with a large change of the picture is a cut.
+Two uses follow:
+
+- **Motion without the camera.** Mask the seconds where the camera moved or cut before
+  summarising quantity of motion, a motiongram or an envelope; otherwise a pan is the
+  biggest "gesture" in the piece.
+- **Counting per framing.** Pass the camera analysis to `performer_count(det, a, b,
+  camera=cam)` and the unit becomes a *framing*, a still run of at least ten seconds. Each
+  framing gets the 75th percentile of its counts and the widest framing is the estimate. On
+  a concert with a moving camera this was exact for a soloist, a duo and a five-piece band
+  where the plain percentile counted the audience in the band's wide shot; a choir stays
+  under-counted because singers occlude each other.
+
+Both analyses cache well: keep the proxy and the detections next to the recording and the
+counts for any span are instant.

mkdocs.yml

@@ -44,6 +44,7 @@ nav:
       - Audio-Video Analysis: user-guide/audio-video.md
       - Sound–Movement Analysis Toolkit: user-guide/sound-movement-toolkit.md
       - Eye Tracking, Events & the Canvas: user-guide/eye-tracking-events-canvas.md
+      - People on Stage & Camera Motion: user-guide/people-and-camera.md
   - Reference:
       - Overview: musicalgestures/index.md
       - Core Classes: user-guide/core-classes.md
@@ -66,6 +67,8 @@ nav:
           - Hierarchy: musicalgestures/_hierarchy.md
           - Tracks: musicalgestures/_tracks.md
           - Room and occupancy: musicalgestures/_plate.md
+          - People on stage: musicalgestures/_performers.md
+          - Camera cuts and PTZ: musicalgestures/_camera.md
       - Annotating & interpreting:
           - Annotate: musicalgestures/_annotate.md
           - Timeline: musicalgestures/_timeline.md

musicalgestures/__init__.py

@@ -88,6 +88,8 @@ def __init__(self):
 
 # --- Sound--motion signal methods (ro / stillstanding / cymbal / Westney studies) ---
 from musicalgestures._peaks import pick_peaks
+from musicalgestures._performers import detect_people, on_stage, people_track, performer_count
+from musicalgestures._camera import camera_motion, camera_state_at, still_runs, make_proxy
 from musicalgestures._laughter import laughter_score, laughter_segments
 from musicalgestures._coaccentuation import (
     co_accentuation,

musicalgestures/_camera.py

@@ -0,0 +1,138 @@
+"""Camera cuts and pan/tilt/zoom, so that camera motion is not read as performer motion.
+
+An operated camera pans, tilts and zooms, and a multi-camera edit cuts between angles. Both
+inflate frame-difference measures (quantity of motion, motiongrams) and change who is in the
+picture. This module labels each sample of a video as ``still``, ``moving`` or ``cut`` from the
+global geometry between consecutive frames: ORB features matched across the pair and a
+partial-affine (translation + scale) RANSAC fit. Consistent geometry with a shift or a scale
+change is a camera move; no consistent geometry together with a large change of the picture
+is a cut; the rest is still. Shots are the spans between cuts; *framings* are the still runs
+between moves and cuts, which is the unit that matters when counting people or comparing
+motion, because the framing is constant inside one.
+
+The analysis runs on a small, low-rate proxy (2 fps, 180 px high by default), made once with
+ffmpeg and reused, so a 90-minute recording takes about a minute.
+"""
+from __future__ import annotations
+
+import json
+import subprocess
+from pathlib import Path
+from typing import cast
+
+import numpy as np
+
+__all__ = ["camera_motion", "camera_state_at", "still_runs", "make_proxy"]
+
+
+def make_proxy(filename: "str | Path", proxy_path: "str | Path", fps: float = 2.0, height: int = 180,
+               ffmpeg_input_args: list[str] | None = None) -> Path:
+    """A low-rate, low-resolution copy of the video (ffmpeg), cached at `proxy_path`.
+    `ffmpeg_input_args` go before ``-i`` (``["-hwaccel", "cuda"]`` decodes on the GPU)."""
+    out: Path = Path(proxy_path)
+    if not out.exists():
+        out.parent.mkdir(parents=True, exist_ok=True)
+        subprocess.run(["ffmpeg", "-v", "error", "-y", *(ffmpeg_input_args or []), "-i", str(filename), "-vf", f"fps={fps},scale=-2:{height}",
+                        "-an", "-c:v", "libx264", "-preset", "veryfast", "-crf", "20", str(out)],
+                       check=True, capture_output=True)
+    return out
+
+
+def camera_motion(filename: "str | Path", proxy_path: "str | Path | None" = None, fps: float = 2.0, height: int = 180, move_px: float = 1.0,
+                  zoom: float = 0.005, min_inliers: int = 15, cache=None, verbose: bool = True,
+                  ffmpeg_input_args: list[str] | None = None) -> dict:
+    """Per-sample camera state for a video.
+
+    Returns a dict with ``hop_s``, ``t`` (sample times), ``state`` (``still`` / ``moving`` / ``cut``),
+    ``tx``, ``ty`` (pixels at proxy scale), ``scale``, ``inliers``, ``cuts`` (times), ``shots``
+    (``{"start", "end"}`` between cuts) and ``summary`` (share of time in each state). `move_px` and
+    `zoom` are the per-sample translation and scale change that count as a move; both were set on an
+    operated concert camera and are conservative for a tripod. Pass `cache` (a JSON path) to reuse.
+    """
+    import cv2
+    if cache and Path(cache).exists():
+        cached: dict = json.loads(Path(cache).read_text())
+        return cached
+    proxy = make_proxy(filename, proxy_path or Path(str(filename)).with_suffix(".camera_proxy.mp4"), fps, height, ffmpeg_input_args)
+    cap = cv2.VideoCapture(str(proxy))
+    real_fps = cap.get(cv2.CAP_PROP_FPS) or fps
+    orb = cv2.ORB_create(nfeatures=400)  # type: ignore[attr-defined]
+    bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
+    prev: tuple | None = None
+    t: list[float] = []; state: list[str] = []; tx: list[float] = []; ty: list[float] = []
+    sc: list[float] = []; inl: list[int] = []
+    i = 0
+    while True:
+        ok, frame = cap.read()
+        if not ok:
+            break
+        g = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
+        kp, des = orb.detectAndCompute(g, None)
+        hist = cv2.calcHist([g], [0], None, [32], [0, 256]); cv2.normalize(hist, hist)
+        if prev is not None:
+            pg, pkp, pdes, phist = prev
+            n_in, dx, dy, s = 0, 0.0, 0.0, 1.0
+            if des is not None and pdes is not None and len(kp) >= 8 and len(pkp) >= 8:
+                m = bf.match(pdes, des)
+                if len(m) >= 8:
+                    src = np.float32([pkp[x.queryIdx].pt for x in m]); dst = np.float32([kp[x.trainIdx].pt for x in m])
+                    A, mask = cv2.estimateAffinePartial2D(src, dst, method=cv2.RANSAC, ransacReprojThreshold=3.0)
+                    if A is not None:
+                        n_in = int(mask.sum()); dx, dy = float(A[0, 2]), float(A[1, 2])
+                        s = float(np.hypot(A[0, 0], A[0, 1]))
+            corr = float(cv2.compareHist(phist, hist, cv2.HISTCMP_CORREL))
+            diff = float(np.abs(g.astype(np.int16) - pg.astype(np.int16)).mean())
+            if n_in < min_inliers and (corr < 0.6 or diff > 30):
+                st = "cut"
+            elif n_in >= min_inliers and (abs(dx) > move_px or abs(dy) > move_px or abs(s - 1) > zoom):
+                st = "moving"
+            else:
+                st = "still"
+            t.append(round(i / real_fps, 3)); state.append(st); tx.append(round(dx, 2)); ty.append(round(dy, 2))
+            sc.append(round(s, 4)); inl.append(n_in)
+        prev = (g, kp, des, hist)
+        i += 1
+        if verbose and i % 2000 == 0:
+            print(f"camera_motion: {i} frames")
+    cap.release()
+    cuts = [t[k] for k in range(len(t)) if state[k] == "cut" and (k == 0 or state[k - 1] != "cut")]
+    edges = [0.0] + cuts + [round(i / real_fps, 3)]
+    out = {"hop_s": round(1 / real_fps, 4), "proxy": str(proxy), "t": t, "state": state, "tx": tx, "ty": ty,
+           "scale": sc, "inliers": inl, "cuts": cuts,
+           "shots": [{"start": a, "end": b} for a, b in zip(edges, edges[1:]) if b - a > 0],
+           "summary": {k: round(state.count(k) / max(1, len(state)), 3) for k in ("still", "moving", "cut")}}
+    if cache:
+        Path(cache).write_text(json.dumps(out))
+    return out
+
+
+def camera_state_at(cam: dict, times) -> np.ndarray:
+    """The camera state at each time (``still`` when the analysis has nothing there)."""
+    times = np.atleast_1d(np.asarray(times, float))
+    states: list[str] = ["still"] * len(times)
+    if cam and cam.get("t"):
+        tt = np.asarray(cam["t"], float)
+        labels: list[str] = list(cam["state"])
+        idx = np.clip(np.searchsorted(tt, times, side="right") - 1, 0, len(tt) - 1)
+        states = [labels[int(i)] for i in idx]
+    return cast("np.ndarray", np.array(states, dtype=object))
+
+
+def still_runs(cam: dict, start_s: float = 0.0, end_s: float | None = None, min_s: float = 10.0) -> list[tuple[float, float]]:
+    """Framings: spans inside ``[start_s, end_s)`` where the camera held still for at least `min_s`."""
+    tt = np.asarray(cam["t"]); st = np.asarray(cam["state"], dtype=object)
+    end_s = float(tt[-1] + cam["hop_s"]) if end_s is None else end_s
+    runs: list[tuple[float, float]] = []
+    a: float | None = None
+    for t, k in zip(tt, st):
+        if t < start_s or t >= end_s:
+            continue
+        if k == "still" and a is None:
+            a = float(t)
+        elif k != "still" and a is not None:
+            if t - a >= min_s:
+                runs.append((a, float(t)))
+            a = None
+    if a is not None and end_s - a >= min_s:
+        runs.append((a, float(end_s)))
+    return runs