Skip to content

Estimate PAM distances with an eigendecomposition of logPAM1 - #30

Merged
alpae merged 2 commits into
DessimozLab:masterfrom
alpae:estimate-pam-eigen
Oct 29, 2026
Merged

alpae merged 2 commits into
DessimozLab:masterfrom
alpae:estimate-pam-eigen

Conversation

@alpae

@alpae alpae commented Oct 29, 2026

Copy link
Copy Markdown
Member

estimate_pam (and with it estimate_many_pam) now uses a new C function EstimatePamEigen instead of EstimatePam:

  • the aligned pairs of symbols are counted once, so scores and likelihoods are sums over the distinct pairs (at most 26²) instead of over all alignment positions;
  • logPAM1 is the rate matrix of a reversible process, so it is decomposed once per MutipleAlEnv (symmetrized with its stationary distribution, then eigh). In the Newton iterations of the variance, exp(pam·logPAM1) and its first two derivatives are then evaluated from the eigenvalues, only for the pairs that occur, instead of with matrix exponentials and products.

Results

  • Score and PAM distance are bit-identical to EstimatePam, and the variance agrees to ≤ 1e-9 relative (worst case at the PAM 1000 saturation). test_darwin passes unchanged.
  • Speed: estimate_pam takes 11 µs instead of 378 µs per pair (300 aa, 30 % substitutions). estimate_many_pam on 100 sequences with 1 thread takes 0.09 s instead of 1.47 s.

Fallback

If a custom log_pam1 is not reversible (its symmetrized form isn't symmetric) or has no positive stationary distribution, MutipleAlEnv falls back to the original EstimatePam. It stays available as the private _estimate_pam_reference for tests.

Tests

test/test_estimate_pam.py compares the new code with the reference on 300 random alignments (10 to 1000 positions, all levels of divergence and gaps), on edge cases (identical sequences, saturated, gaps only, X, B/Z), and through estimate_many_pam. It also checks the fallback with a perturbed non-reversible logPAM1.

alpae added 2 commits October 29, 2026 21:52
EstimatePam spends most of its time computing exp(pam * logPAM1) and its
derivatives with matrix exponentials and products in every Newton
iteration of the variance estimation, and summing scores over all
alignment positions.

EstimatePamEigen counts the distinct aligned pairs of symbols once and
evaluates scores and likelihoods over these pairs. logPAM1 is the rate
matrix of a reversible process, so it is decomposed once per
MutipleAlEnv (symmetrized with its stationary distribution, then eigh),
and the needed entries of exp(pam * logPAM1) and its derivatives are
computed from the eigenvalues only for the pairs that occur.

Scores and PAM distances are identical, variances agree to about 1e-9
relative. estimate_pam is about 30x faster per pair.

If a custom logPAM1 is not reversible, estimate_pam and
estimate_many_pam fall back to the original EstimatePam, which stays
available as _estimate_pam_reference for tests.
@alpae
alpae force-pushed the estimate-pam-eigen branch from 0f04217 to 1e4bb80 Compare October 29, 2026 20:53
@alpae
alpae merged commit e723eae into DessimozLab:master Oct 29, 2026
19 checks passed
@alpae
alpae deleted the estimate-pam-eigen branch October 29, 2026 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant