From d5e8b789c4e66c3e08536e2b3d4c8edd2a02cecb Mon Sep 17 00:00:00 2001 From: Heberto Mayorquin Date: Fri, 22 May 2026 10:27:14 -0600 Subject: [PATCH 1/8] add catalogue-recording pattern documentation --- doc/neuropixels_readers.rst | 176 ++++++++++++++++++++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 doc/neuropixels_readers.rst diff --git a/doc/neuropixels_readers.rst b/doc/neuropixels_readers.rst new file mode 100644 index 00000000..f7c62a37 --- /dev/null +++ b/doc/neuropixels_readers.rst @@ -0,0 +1,176 @@ +Neuropixels format readers: catalogue construction and recording-specific wiring +================================================================================= + +.. currentmodule:: probeinterface + + +The catalogue: :py:func:`build_neuropixels_probe` +------------------------------------------------- + +The foundation of every Neuropixels reader in probeinterface is +:py:func:`build_neuropixels_probe`. Given a probe part number (a specific +stock-keeping unit identifier such as ``"NP1000"``, ``"NP2000"``, ``"NP2014"``), +it returns a :py:class:`Probe` carrying the full silicon geometry for that +part number: every catalogue contact (960 for Neuropixels 1.0, 1280 per shank +for Neuropixels 2.0), the planar contour of the shanks, the contact shapes and +sizes, the analog-to-digital converter (ADC) multiplexer (MUX) routing table, +and the probe-level annotations (``manufacturer``, ``model_name``, +``part_number``, ``description``). + +The numbers behind that geometry come from the +`ProbeTable `_ repository maintained +by `Bill Karsh `_ (author of SpikeGLX). +ProbeTable is the canonical machine-readable inventory of IMEC Neuropixels +probe specifications: contact positions, electrode dimensions, shank geometry, +MUX routing, ADC configuration, all keyed by part number. Probeinterface +mirrors a postprocessed snapshot of that data into the package via +``resources/postprocess_neuropixels_probe_features.py``, which is re-run after +each ProbeTable sync. Without ProbeTable, every reader would have to carry +its own hand-written copy of the manufacturer specs, which is exactly the +situation the catalogue pattern is designed to avoid. + + +The format readers +------------------ + +Four entry points read Neuropixels recordings (or recording configurations) +and produce a probe ready to use with SpikeInterface: + +.. list-table:: + :header-rows: 1 + :widths: 30 30 40 + + * - Reader + - Input + - Where the part number comes from + * - :py:func:`read_spikeglx` + - SpikeGLX ``.ap.meta`` (plus the ``.ap.bin`` it describes) + - ``imDatPrb_pn`` field in the meta file + * - :py:func:`read_openephys_neuropixels` + - Open Ephys ``settings.xml`` (plus the binary stream it describes) + - ``probe_part_number`` attribute in the XML + * - :py:func:`read_imro` + - SpikeGLX IMRO (Imec ReadOut) table file (``.imro``) + - First field of the IMRO header: a part number directly (new SpikeGLX + format) or a legacy numeric type code translated to a part number via + the catalogue mapping (old format). See SpikeGLX issue + `#432 `_ + for the format transition. + * - :py:func:`read_spikegadgets_neuropixels` + - SpikeGadgets ``.rec`` XML header + - Not present in the file; the reader picks a geometry-equivalent stand-in + based on ``(SpikeConfiguration.device, deviceSubType)``: ``NP1000`` for + Neuropixels 1.0, ``NP2000`` for Neuropixels 2.0 single-shank, ``NP2014`` + for Neuropixels 2.0 4-shank + +The first three readers identify the actual probe stock-keeping unit (SKU) +from the recording metadata. SpikeGadgets is the exception: its ``.rec`` XML +does not carry a part number field, so the reader cannot identify the SKU. +It picks one representative per geometry-equivalent family (all Neuropixels +1.0 staggered variants share contact positions; all Neuropixels 2.0 +single-shank variants share contact positions; all Neuropixels 2.0 4-shank +variants share contact positions) and clears the ``model_name``, +``description``, and ``part_number`` annotations on the returned probe so +downstream code does not read the stand-in as an attribution. + + +From catalogue probe to probe in a recording setup +-------------------------------------------------- + +The catalogue probe is pure geometry, divorced from any session. A real +recording uses only a subset of those contacts: the Neuropixels headstage +acquires 384 channels at a time, and the recording configuration selects +which catalogue contacts those 384 are drawn from (384 of 960 on Neuropixels +1.0, 384 of 1280 per shank on Neuropixels 2.0 single-shank, 384 of 5120 on +Neuropixels 2.0 4-shank). The selection mechanism differs by recording +format (an IMRO table for SpikeGLX, a channel map in ``settings.xml`` for +Open Ephys, the ``SpikeNTrode`` list in SpikeGadgets's ``.rec`` XML); the +"Matching catalogue contacts to recorded data" section below covers each +case. On top of the selection, the recording adds session-specific state: +per-contact analog band (AP) and local field potential (LFP) gains, ADC +sample order, reference configuration, and the channel-to-file mapping that +says where each contact's data lives in the saved binary. Probeinterface +calls the result a probe in a recording setup, to distinguish it from the +catalogue. + +Each reader produces the recording-setup probe in the same three steps: + +1. Build the catalogue probe by calling + :py:func:`build_neuropixels_probe(part_number) ` + with the part number obtained from the recording metadata. +2. Slice the catalogue probe to the active electrodes for this session via + :py:meth:`probe.get_slice(active_indices) `. The slice + drops the unrecorded contacts but preserves the probe-level annotations + and the per-contact catalogue annotations (ADC group, sample order) on the + contacts that survive. +3. Attach the recording-specific state: per-contact AP/LFP gains, any + reference annotations, and finally + :py:meth:`probe.set_device_channel_indices(...) ` + to record where each surviving contact's data lives in the saved file. + + +Matching catalogue contacts to recorded data +-------------------------------------------- + +The catalogue-build step is the same across readers; the matching step is +where the formats differ. ``active_indices`` and ``device_channel_indices`` +come from a different field in each metadata source: + +* **SpikeGLX:** ``active_indices`` is the electrode list parsed from the IMRO + table embedded in the ``.ap.meta`` file. ``device_channel_indices`` is + identity (``np.arange(n)``) because SpikeGLX writes one column per active + electrode in IMRO selection order. +* **Open Ephys:** ``active_indices`` is the electrode list parsed from the + ``CHANNELS`` block in ``settings.xml``. ``device_channel_indices`` follows + the order the binary stream uses, which the same XML file describes. +* **IMRO (standalone):** ``active_indices`` is parsed directly from the IMRO + entries. There is no recording to wire to, so :py:func:`read_imro` returns + the sliced probe without setting ``device_channel_indices``; callers that + have a corresponding ``.ap.bin`` use :py:func:`read_spikeglx` instead. +* **SpikeGadgets:** ``active_indices`` is the list of ``SpikeNTrode`` + electrodes from the ``.rec`` XML, remapped from Trodes' ``channelsOn`` bit + order to the catalogue's contact order (an identity remap for Neuropixels + 1.0; a row-major-to-shank-major remap for Neuropixels 2.0 4-shank; a + per-row column swap for Neuropixels 2.0 single-shank). + ``device_channel_indices`` is the ``hwChan`` attribute on each + ``SpikeNTrode``, which happens to coincide with the column index in the + SpikeGadgets datalogger's binary stream because the firmware writes samples + in ``hwChan`` ascending order. + + +What the pattern solves +----------------------- + +Constructing geometry from scratch inside each format reader (the situation +before the catalogue pattern) had three problems: + +* **Geometry drift across readers.** Each reader carried its own copy of the + manufacturer specs. A Neuropixels 2.0 4-shank probe loaded through SpikeGLX + and through SpikeGadgets could return contact positions that disagreed in + the third decimal because the two readers had been updated against + different snapshots of the IMEC spec. Centralising the geometry in + :py:func:`build_neuropixels_probe` and sourcing it from ProbeTable means + every reader returns the same positions for the same part number. +* **Conflated geometry and wiring bugs.** When a saved recording looked wrong + on the probe, it was difficult to say whether the geometry was off + (catalogue issue) or the channel-to-contact mapping was off (wiring issue). + With the two phases separated, a geometry bug is a bug in + :py:func:`build_neuropixels_probe`; a wiring bug is a bug in the reader's + matching step. The two can be diagnosed and fixed independently. +* **Hidden active-electrode selection.** Readers that built a 384-contact + probe directly hid the fact that 576 catalogue contacts were silently + dropped. The explicit ``probe.get_slice(active_indices)`` step makes the + selection visible and inspectable: callers can ask "which catalogue + contacts did this session record?" and get a direct answer. + +The pattern also pays out on the upgrade path. When IMEC ships a new probe +variant, the integration work is "add the part number to ProbeTable, re-run +the postprocess script". + + +Discussion +---------- + +This pattern was proposed and is tracked in issue +`#405 `_; if you have +any discussion point to add please re-open the issue so the mantainers can disucss. From f879ed0370966d205f985f291831abeddf7b1fa5 Mon Sep 17 00:00:00 2001 From: Heberto Mayorquin Date: Fri, 22 May 2026 10:27:29 -0600 Subject: [PATCH 2/8] document --- doc/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/index.rst b/doc/index.rst index aed06dfb..9ca01ef2 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -34,5 +34,6 @@ Here is a schema for the naming used in the package: examples/index.rst format_spec library + neuropixels_readers api release_notes From a4a0d01cecb26cf3285214456eb14ebb865d6f9f Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Fri, 22 May 2026 16:30:01 +0000 Subject: [PATCH 3/8] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- doc/neuropixels_readers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/neuropixels_readers.rst b/doc/neuropixels_readers.rst index f7c62a37..1886d46a 100644 --- a/doc/neuropixels_readers.rst +++ b/doc/neuropixels_readers.rst @@ -165,7 +165,7 @@ before the catalogue pattern) had three problems: The pattern also pays out on the upgrade path. When IMEC ships a new probe variant, the integration work is "add the part number to ProbeTable, re-run -the postprocess script". +the postprocess script". Discussion From 5db34fdbf80bf2d128e94076c6e46c7afcb32af2 Mon Sep 17 00:00:00 2001 From: Heberto Mayorquin Date: Fri, 22 May 2026 10:32:45 -0600 Subject: [PATCH 4/8] spelling --- doc/neuropixels_readers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/neuropixels_readers.rst b/doc/neuropixels_readers.rst index f7c62a37..fa5c0983 100644 --- a/doc/neuropixels_readers.rst +++ b/doc/neuropixels_readers.rst @@ -173,4 +173,4 @@ Discussion This pattern was proposed and is tracked in issue `#405 `_; if you have -any discussion point to add please re-open the issue so the mantainers can disucss. +any discussion point to add please re-open the issue so the maintainers can discuss. From 852fc67a2aaf402012ee49f4d47e76265f5266cb Mon Sep 17 00:00:00 2001 From: Heberto Mayorquin Date: Fri, 22 May 2026 10:59:00 -0600 Subject: [PATCH 5/8] title --- doc/neuropixels_readers.rst | 64 +++++++++++-------------------------- 1 file changed, 18 insertions(+), 46 deletions(-) diff --git a/doc/neuropixels_readers.rst b/doc/neuropixels_readers.rst index 131755cf..29d9ad10 100644 --- a/doc/neuropixels_readers.rst +++ b/doc/neuropixels_readers.rst @@ -1,5 +1,5 @@ -Neuropixels format readers: catalogue construction and recording-specific wiring -================================================================================= +The Neuropixels catalogue pattern +================================= .. currentmodule:: probeinterface @@ -63,30 +63,31 @@ and produce a probe ready to use with SpikeInterface: Neuropixels 1.0, ``NP2000`` for Neuropixels 2.0 single-shank, ``NP2014`` for Neuropixels 2.0 4-shank -The first three readers identify the actual probe stock-keeping unit (SKU) -from the recording metadata. SpikeGadgets is the exception: its ``.rec`` XML -does not carry a part number field, so the reader cannot identify the SKU. -It picks one representative per geometry-equivalent family (all Neuropixels -1.0 staggered variants share contact positions; all Neuropixels 2.0 -single-shank variants share contact positions; all Neuropixels 2.0 4-shank -variants share contact positions) and clears the ``model_name``, -``description``, and ``part_number`` annotations on the returned probe so -downstream code does not read the stand-in as an attribution. +The first three readers read the part number directly from the recording +metadata. SpikeGadgets is the exception: its ``.rec`` XML does not carry a +part number field, so the reader cannot know which specific variant produced +the recording. It picks one representative per geometry-equivalent family +(all Neuropixels 1.0 staggered variants share contact positions; all +Neuropixels 2.0 single-shank variants share contact positions; all +Neuropixels 2.0 4-shank variants share contact positions) and clears the +``model_name``, ``description``, and ``part_number`` annotations on the +returned probe so downstream code does not read the stand-in as an +attribution. From catalogue probe to probe in a recording setup -------------------------------------------------- -The catalogue probe is pure geometry, divorced from any session. A real +The catalogue probe is pure geometry, divorced from any recording session. A real recording uses only a subset of those contacts: the Neuropixels headstage acquires 384 channels at a time, and the recording configuration selects which catalogue contacts those 384 are drawn from (384 of 960 on Neuropixels 1.0, 384 of 1280 per shank on Neuropixels 2.0 single-shank, 384 of 5120 on Neuropixels 2.0 4-shank). The selection mechanism differs by recording format (an IMRO table for SpikeGLX, a channel map in ``settings.xml`` for -Open Ephys, the ``SpikeNTrode`` list in SpikeGadgets's ``.rec`` XML); the -"Matching catalogue contacts to recorded data" section below covers each -case. On top of the selection, the recording adds session-specific state: +Open Ephys, the ``SpikeNTrode`` list in SpikeGadgets's ``.rec`` XML); each +reader's docstring covers the specifics for that format. On top of the +selection, the recording adds session-specific state: per-contact analog band (AP) and local field potential (LFP) gains, ADC sample order, reference configuration, and the channel-to-file mapping that says where each contact's data lives in the saved binary. Probeinterface @@ -98,7 +99,7 @@ Each reader produces the recording-setup probe in the same three steps: 1. Build the catalogue probe by calling :py:func:`build_neuropixels_probe(part_number) ` with the part number obtained from the recording metadata. -2. Slice the catalogue probe to the active electrodes for this session via +2. Slice the catalogue probe to the active electrodes for this recording session via :py:meth:`probe.get_slice(active_indices) `. The slice drops the unrecorded contacts but preserves the probe-level annotations and the per-contact catalogue annotations (ADC group, sample order) on the @@ -109,35 +110,6 @@ Each reader produces the recording-setup probe in the same three steps: to record where each surviving contact's data lives in the saved file. -Matching catalogue contacts to recorded data --------------------------------------------- - -The catalogue-build step is the same across readers; the matching step is -where the formats differ. ``active_indices`` and ``device_channel_indices`` -come from a different field in each metadata source: - -* **SpikeGLX:** ``active_indices`` is the electrode list parsed from the IMRO - table embedded in the ``.ap.meta`` file. ``device_channel_indices`` is - identity (``np.arange(n)``) because SpikeGLX writes one column per active - electrode in IMRO selection order. -* **Open Ephys:** ``active_indices`` is the electrode list parsed from the - ``CHANNELS`` block in ``settings.xml``. ``device_channel_indices`` follows - the order the binary stream uses, which the same XML file describes. -* **IMRO (standalone):** ``active_indices`` is parsed directly from the IMRO - entries. There is no recording to wire to, so :py:func:`read_imro` returns - the sliced probe without setting ``device_channel_indices``; callers that - have a corresponding ``.ap.bin`` use :py:func:`read_spikeglx` instead. -* **SpikeGadgets:** ``active_indices`` is the list of ``SpikeNTrode`` - electrodes from the ``.rec`` XML, remapped from Trodes' ``channelsOn`` bit - order to the catalogue's contact order (an identity remap for Neuropixels - 1.0; a row-major-to-shank-major remap for Neuropixels 2.0 4-shank; a - per-row column swap for Neuropixels 2.0 single-shank). - ``device_channel_indices`` is the ``hwChan`` attribute on each - ``SpikeNTrode``, which happens to coincide with the column index in the - SpikeGadgets datalogger's binary stream because the firmware writes samples - in ``hwChan`` ascending order. - - What the pattern solves ----------------------- @@ -161,7 +133,7 @@ before the catalogue pattern) had three problems: probe directly hid the fact that 576 catalogue contacts were silently dropped. The explicit ``probe.get_slice(active_indices)`` step makes the selection visible and inspectable: callers can ask "which catalogue - contacts did this session record?" and get a direct answer. + contacts did this recording session record?" and get a direct answer. The pattern also pays out on the upgrade path. When IMEC ships a new probe variant, the integration work is "add the part number to ProbeTable, re-run From ce65555ae735b7c0f7bda915a706a4a5630ca947 Mon Sep 17 00:00:00 2001 From: Heberto Mayorquin Date: Fri, 22 May 2026 11:20:14 -0600 Subject: [PATCH 6/8] bette structure --- .gitignore | 2 + doc/neuropixels_readers.rst | 207 ++++++++++++++++++------------------ 2 files changed, 107 insertions(+), 102 deletions(-) diff --git a/.gitignore b/.gitignore index 52a45ce4..695ffc81 100644 --- a/.gitignore +++ b/.gitignore @@ -11,6 +11,7 @@ tests/*.h5 examples/*.prb examples/*.h5 +examples/*.json ressources/*/* build/* @@ -19,6 +20,7 @@ build/* dist/* doc/_* doc/examples/* +doc/sg_execution_times.rst dev_* .coverage diff --git a/doc/neuropixels_readers.rst b/doc/neuropixels_readers.rst index 29d9ad10..a5c59a7f 100644 --- a/doc/neuropixels_readers.rst +++ b/doc/neuropixels_readers.rst @@ -4,37 +4,84 @@ The Neuropixels catalogue pattern .. currentmodule:: probeinterface -The catalogue: :py:func:`build_neuropixels_probe` -------------------------------------------------- - -The foundation of every Neuropixels reader in probeinterface is -:py:func:`build_neuropixels_probe`. Given a probe part number (a specific -stock-keeping unit identifier such as ``"NP1000"``, ``"NP2000"``, ``"NP2014"``), -it returns a :py:class:`Probe` carrying the full silicon geometry for that -part number: every catalogue contact (960 for Neuropixels 1.0, 1280 per shank -for Neuropixels 2.0), the planar contour of the shanks, the contact shapes and -sizes, the analog-to-digital converter (ADC) multiplexer (MUX) routing table, -and the probe-level annotations (``manufacturer``, ``model_name``, -``part_number``, ``description``). - -The numbers behind that geometry come from the +Two kinds of Neuropixels probe +------------------------------ + +Probeinterface distinguishes two kinds of Neuropixels probe. + +**Catalogue probe.** The probe as it appears in the IMEC catalogue, with +every contact on the silicon present (960 on Neuropixels 1.0, 1280 per +shank on Neuropixels 2.0). Built via :py:func:`build_neuropixels_probe(part_number) +`. It carries: + +* contact positions +* shank contour and dimensions +* contact shapes and sizes +* the analog-to-digital converter (ADC) multiplexer (MUX) routing on the + silicon +* identity metadata (manufacturer, model name, part number, description) + +A catalogue probe is pure geometry, the same for every recording made with +that variant. Use it to plot the probe layout, compute distances between +contacts, or run any analysis that does not depend on a specific recording. + +**Recording-setup probe.** The catalogue probe specialised for one +recording session: only the contacts actually recorded are present +(typically 384 of the 960 or more catalogue contacts), and per-contact +recording state is attached: + +* per-contact analog band (AP) and local field potential (LFP) gains +* reference configuration +* per-contact sampling order +* probe wiring (the mapping from each contact to the recording channel + that captured its data) + +Use a recording-setup probe to hand the recording to SpikeInterface so +spike sorters see both the geometry and the correct channel mapping, to +convert raw samples to microvolts via the per-contact gains, or to plot +the recorded contacts alongside the recorded traces. + + +How readers connect the two +--------------------------- + +A format reader turns a Neuropixels recording into a recording-setup probe +in three steps: + +1. **Fetch the catalogue probe.** Look up the probe part number (SKU) in + the recording's metadata, then call + :py:func:`build_neuropixels_probe(part_number) `. +2. **Identify the active electrodes.** Read the recording's channel + configuration to find which catalogue contacts were actually recorded, + and slice the catalogue probe down to that subset via + :py:meth:`probe.get_slice(active_indices) `. +3. **Attach the recording-setup metadata.** Add the per-contact gains, + reference settings, and sampling order, and set the probe wiring. + +The sections below cover where the part number comes from per reader +(step 1) and where the catalogue data itself comes from. + + +The catalogue +------------- + +A probe part number (SKU) is an IMEC identifier such as ``"NP1000"``, +``"NP2000"``, or ``"NP2014"``. The part number determines the silicon +geometry: 960 contacts on Neuropixels 1.0, 1280 per shank on Neuropixels +2.0, plus all the per-variant pitch and shank dimensions. + +The data behind the catalogue comes from the `ProbeTable `_ repository maintained by `Bill Karsh `_ (author of SpikeGLX). ProbeTable is the canonical machine-readable inventory of IMEC Neuropixels -probe specifications: contact positions, electrode dimensions, shank geometry, -MUX routing, ADC configuration, all keyed by part number. Probeinterface -mirrors a postprocessed snapshot of that data into the package via -``resources/postprocess_neuropixels_probe_features.py``, which is re-run after -each ProbeTable sync. Without ProbeTable, every reader would have to carry -its own hand-written copy of the manufacturer specs, which is exactly the -situation the catalogue pattern is designed to avoid. +probe specifications, all keyed by part number. -The format readers ------------------- +The readers +----------- -Four entry points read Neuropixels recordings (or recording configurations) -and produce a probe ready to use with SpikeInterface: +Four readers produce a probe from a Neuropixels recording. They differ in +where they look up the part number: .. list-table:: :header-rows: 1 @@ -58,91 +105,47 @@ and produce a probe ready to use with SpikeInterface: for the format transition. * - :py:func:`read_spikegadgets_neuropixels` - SpikeGadgets ``.rec`` XML header - - Not present in the file; the reader picks a geometry-equivalent stand-in - based on ``(SpikeConfiguration.device, deviceSubType)``: ``NP1000`` for - Neuropixels 1.0, ``NP2000`` for Neuropixels 2.0 single-shank, ``NP2014`` - for Neuropixels 2.0 4-shank - -The first three readers read the part number directly from the recording -metadata. SpikeGadgets is the exception: its ``.rec`` XML does not carry a -part number field, so the reader cannot know which specific variant produced -the recording. It picks one representative per geometry-equivalent family -(all Neuropixels 1.0 staggered variants share contact positions; all -Neuropixels 2.0 single-shank variants share contact positions; all -Neuropixels 2.0 4-shank variants share contact positions) and clears the -``model_name``, ``description``, and ``part_number`` annotations on the -returned probe so downstream code does not read the stand-in as an -attribution. - - -From catalogue probe to probe in a recording setup --------------------------------------------------- - -The catalogue probe is pure geometry, divorced from any recording session. A real -recording uses only a subset of those contacts: the Neuropixels headstage -acquires 384 channels at a time, and the recording configuration selects -which catalogue contacts those 384 are drawn from (384 of 960 on Neuropixels -1.0, 384 of 1280 per shank on Neuropixels 2.0 single-shank, 384 of 5120 on -Neuropixels 2.0 4-shank). The selection mechanism differs by recording -format (an IMRO table for SpikeGLX, a channel map in ``settings.xml`` for -Open Ephys, the ``SpikeNTrode`` list in SpikeGadgets's ``.rec`` XML); each -reader's docstring covers the specifics for that format. On top of the -selection, the recording adds session-specific state: -per-contact analog band (AP) and local field potential (LFP) gains, ADC -sample order, reference configuration, and the channel-to-file mapping that -says where each contact's data lives in the saved binary. Probeinterface -calls the result a probe in a recording setup, to distinguish it from the -catalogue. - -Each reader produces the recording-setup probe in the same three steps: - -1. Build the catalogue probe by calling - :py:func:`build_neuropixels_probe(part_number) ` - with the part number obtained from the recording metadata. -2. Slice the catalogue probe to the active electrodes for this recording session via - :py:meth:`probe.get_slice(active_indices) `. The slice - drops the unrecorded contacts but preserves the probe-level annotations - and the per-contact catalogue annotations (ADC group, sample order) on the - contacts that survive. -3. Attach the recording-specific state: per-contact AP/LFP gains, any - reference annotations, and finally - :py:meth:`probe.set_device_channel_indices(...) ` - to record where each surviving contact's data lives in the saved file. + - Not present in the file; the reader picks a geometry-equivalent + stand-in based on ``(SpikeConfiguration.device, deviceSubType)``: + ``NP1000`` for Neuropixels 1.0, ``NP2000`` for Neuropixels 2.0 + single-shank, ``NP2014`` for Neuropixels 2.0 4-shank + +The first three readers read the part number directly. SpikeGadgets is the +exception (its ``.rec`` XML does not carry a part number) and falls back to +a geometry-equivalent stand-in; the variants within each Neuropixels family +share identical 2D contact geometry, so any representative produces correct +positions. What the pattern solves ----------------------- -Constructing geometry from scratch inside each format reader (the situation -before the catalogue pattern) had three problems: - -* **Geometry drift across readers.** Each reader carried its own copy of the - manufacturer specs. A Neuropixels 2.0 4-shank probe loaded through SpikeGLX - and through SpikeGadgets could return contact positions that disagreed in - the third decimal because the two readers had been updated against - different snapshots of the IMEC spec. Centralising the geometry in - :py:func:`build_neuropixels_probe` and sourcing it from ProbeTable means - every reader returns the same positions for the same part number. -* **Conflated geometry and wiring bugs.** When a saved recording looked wrong - on the probe, it was difficult to say whether the geometry was off - (catalogue issue) or the channel-to-contact mapping was off (wiring issue). - With the two phases separated, a geometry bug is a bug in - :py:func:`build_neuropixels_probe`; a wiring bug is a bug in the reader's - matching step. The two can be diagnosed and fixed independently. -* **Hidden active-electrode selection.** Readers that built a 384-contact - probe directly hid the fact that 576 catalogue contacts were silently - dropped. The explicit ``probe.get_slice(active_indices)`` step makes the - selection visible and inspectable: callers can ask "which catalogue - contacts did this recording session record?" and get a direct answer. - -The pattern also pays out on the upgrade path. When IMEC ships a new probe -variant, the integration work is "add the part number to ProbeTable, re-run -the postprocess script". +Building geometry from scratch inside each reader (the situation before this +pattern) caused three problems: + +* **Drift across readers.** Each reader carried its own copy of the + manufacturer specs, and the copies could disagree. Centralising the + geometry in :py:func:`build_neuropixels_probe` and sourcing it from + ProbeTable means every reader returns the same positions for the same + part number. +* **Confused bugs.** When a saved recording looked wrong on the probe, it + was hard to tell whether the geometry was off or the channel-to-contact + mapping was off. With the two phases separated, a geometry bug is a bug + in :py:func:`build_neuropixels_probe`; a wiring bug is a bug in the + reader's slicing or wiring step. The two can be diagnosed independently. +* **Hidden electrode selection.** A reader that built a 384-contact probe + directly hid the fact that 576 catalogue contacts were silently dropped. + The explicit slice step makes the selection visible: callers can ask + which catalogue contacts the recording captured and get a direct answer. + +The pattern also helps on the upgrade path. When IMEC ships a new probe +variant, the integration work is to add the part number to ProbeTable; the +readers do not change. Discussion ---------- This pattern was proposed and is tracked in issue -`#405 `_; if you have -any discussion point to add please re-open the issue so the maintainers can discuss. +`#405 `_. +Reopen the issue if you want to discuss changes. From 1c77f7238f55223fbe1fcff90b7fa8b920bf95c9 Mon Sep 17 00:00:00 2001 From: Heberto Mayorquin Date: Fri, 22 May 2026 11:22:29 -0600 Subject: [PATCH 7/8] perfect muack done --- doc/neuropixels_readers.rst | 45 ++++++++++++++++--------------------- 1 file changed, 19 insertions(+), 26 deletions(-) diff --git a/doc/neuropixels_readers.rst b/doc/neuropixels_readers.rst index a5c59a7f..2d560b3a 100644 --- a/doc/neuropixels_readers.rst +++ b/doc/neuropixels_readers.rst @@ -42,28 +42,8 @@ convert raw samples to microvolts via the per-contact gains, or to plot the recorded contacts alongside the recorded traces. -How readers connect the two ---------------------------- - -A format reader turns a Neuropixels recording into a recording-setup probe -in three steps: - -1. **Fetch the catalogue probe.** Look up the probe part number (SKU) in - the recording's metadata, then call - :py:func:`build_neuropixels_probe(part_number) `. -2. **Identify the active electrodes.** Read the recording's channel - configuration to find which catalogue contacts were actually recorded, - and slice the catalogue probe down to that subset via - :py:meth:`probe.get_slice(active_indices) `. -3. **Attach the recording-setup metadata.** Add the per-contact gains, - reference settings, and sampling order, and set the probe wiring. - -The sections below cover where the part number comes from per reader -(step 1) and where the catalogue data itself comes from. - - -The catalogue -------------- +The catalogue source +-------------------- A probe part number (SKU) is an IMEC identifier such as ``"NP1000"``, ``"NP2000"``, or ``"NP2014"``. The part number determines the silicon @@ -77,11 +57,24 @@ ProbeTable is the canonical machine-readable inventory of IMEC Neuropixels probe specifications, all keyed by part number. -The readers ------------ +Format readers +-------------- + +A format reader turns a Neuropixels recording into a recording-setup probe +in three steps: + +1. **Fetch the catalogue probe.** Look up the probe part number (SKU) in + the recording's metadata, then call + :py:func:`build_neuropixels_probe(part_number) `. +2. **Identify the active electrodes.** Read the recording's channel + configuration to find which catalogue contacts were actually recorded, + and slice the catalogue probe down to that subset via + :py:meth:`probe.get_slice(active_indices) `. +3. **Attach the recording-setup metadata.** Add the per-contact gains, + reference settings, and sampling order, and set the probe wiring. -Four readers produce a probe from a Neuropixels recording. They differ in -where they look up the part number: +The four readers in probeinterface differ in step 1: where they look up the +part number in the recording's metadata. .. list-table:: :header-rows: 1 From c2bb5b5b708ca46053d91243b204a07bc0c80858 Mon Sep 17 00:00:00 2001 From: Alessio Buccino Date: Mon, 8 Jun 2026 15:33:07 +0200 Subject: [PATCH 8/8] Apply suggestion from @alejoe91 --- doc/neuropixels_readers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/neuropixels_readers.rst b/doc/neuropixels_readers.rst index 2d560b3a..ba400ed3 100644 --- a/doc/neuropixels_readers.rst +++ b/doc/neuropixels_readers.rst @@ -25,7 +25,7 @@ A catalogue probe is pure geometry, the same for every recording made with that variant. Use it to plot the probe layout, compute distances between contacts, or run any analysis that does not depend on a specific recording. -**Recording-setup probe.** The catalogue probe specialised for one +**Recording-setup probe.** The catalogue probe specialized for one recording session: only the contacts actually recorded are present (typically 384 of the 960 or more catalogue contacts), and per-contact recording state is attached: