Skip to content

Fix FactoredMatrix indexing returning empty result for -1 index#1340

Merged
jlarson4 merged 7 commits into
TransformerLensOrg:devfrom
Kymi808:fix/factored-matrix-negative-index
May 28, 2026
Merged

Fix FactoredMatrix indexing returning empty result for -1 index#1340
jlarson4 merged 7 commits into
TransformerLensOrg:devfrom
Kymi808:fix/factored-matrix-negative-index

Conversation

@Kymi808
Copy link
Copy Markdown

@Kymi808 Kymi808 commented May 28, 2026

Description

FactoredMatrix.__getitem__ routes an integer index into the matrix (ldim/rdim) dimensions through _convert_to_slice, which turns an int v into slice(v, v + 1). For v == -1 this becomes slice(-1, 0) — an empty slice — so indexing the last row/column with a negative index silently returns a (0, ...) tensor instead of the requested element.

import torch
from transformer_lens import FactoredMatrix

fm = FactoredMatrix(torch.rand(2, 4, 3), torch.rand(2, 3, 5))  # shape (2, 4, 5)

fm[0, -1].AB.squeeze().shape      # got (0, 5), expected (5,)
fm[0, 1, -1].AB.squeeze().shape   # got (0,),  expected ()

Only -1 is affected; other negative indices (-2, -3, …) work because v + 1 stays negative and produces a valid slice.

Fix

In _convert_to_slice, use None as the slice stop when v == -1, so slice(-1, None) keeps the final element. Positive indices and other negatives are unchanged.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective (negative indices on each matrix dimension in tests/unit/factored_matrix/test_get_item.py; they fail on main and pass with this change)
  • New and existing unit tests pass locally with my changes (pytest tests/unit/factored_matrix/ → 147 passed, 1 skipped)
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

brendanlong and others added 7 commits April 20, 2026 14:50
* Fix type of HookedTransformerConfig.device

This is typed as `Optional[str]` but sometimes returns `torch.device`.
Updated the code to just return the `str` instead of wrapping with a
device.

I'm not confident that every function which takes a device will
always be passed a string, so I didn't change functions like
warn_if_mps.

Found while working on TransformerLensOrg#1219

* more cleanup

* 3.0 CI Bugs (TransformerLensOrg#1261)

* Fixing `utils` imports

* skip gated notebooks on PR from forks

* Updating notebooks

* Ensure LLaMA only runs when HF_TOKEN is available

---------

Co-authored-by: jlarson4 <jonahalarson@comcast.net>
* Fix TransformerBridge backward hook cleanup

* Preserve backward hooks in run_with_cache
FactoredMatrix.__getitem__ converts an integer index `v` into the matrix
(ldim/rdim) dimensions to `slice(v, v + 1)`. For `v == -1` this becomes
`slice(-1, 0)`, which is an empty slice, so indexing the last row/column
with a negative index silently returns a (0, ...) tensor instead of the
requested element. Other negative indices (-2, -3, ...) are unaffected
because `v + 1` stays negative.

Use `None` as the slice stop when `v == -1` so the final element is kept.
Adds regression tests covering negative indices on each matrix dimension.
@jlarson4 jlarson4 changed the base branch from main to dev May 28, 2026 13:59
@jlarson4
Copy link
Copy Markdown
Collaborator

Excellent work on this @Kymi808! Thanks for your contribution, merging now

@jlarson4 jlarson4 merged commit b3b9934 into TransformerLensOrg:dev May 28, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants