Repository navigation
Estimate PAM distances with an eigendecomposition of logPAM1 - #30
Merged
Merged
Conversation
EstimatePam spends most of its time computing exp(pam * logPAM1) and its derivatives with matrix exponentials and products in every Newton iteration of the variance estimation, and summing scores over all alignment positions. EstimatePamEigen counts the distinct aligned pairs of symbols once and evaluates scores and likelihoods over these pairs. logPAM1 is the rate matrix of a reversible process, so it is decomposed once per MutipleAlEnv (symmetrized with its stationary distribution, then eigh), and the needed entries of exp(pam * logPAM1) and its derivatives are computed from the eigenvalues only for the pairs that occur. Scores and PAM distances are identical, variances agree to about 1e-9 relative. estimate_pam is about 30x faster per pair. If a custom logPAM1 is not reversible, estimate_pam and estimate_many_pam fall back to the original EstimatePam, which stays available as _estimate_pam_reference for tests.
alpae
force-pushed
the
estimate-pam-eigen
branch
from
October 29, 2026 20:53
0f04217 to
1e4bb80
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
estimate_pam(and with itestimate_many_pam) now uses a new C functionEstimatePamEigeninstead ofEstimatePam:MutipleAlEnv(symmetrized with its stationary distribution, theneigh). In the Newton iterations of the variance,exp(pam·logPAM1)and its first two derivatives are then evaluated from the eigenvalues, only for the pairs that occur, instead of with matrix exponentials and products.Results
EstimatePam, and the variance agrees to ≤ 1e-9 relative (worst case at the PAM 1000 saturation).test_darwinpasses unchanged.estimate_pamtakes 11 µs instead of 378 µs per pair (300 aa, 30 % substitutions).estimate_many_pamon 100 sequences with 1 thread takes 0.09 s instead of 1.47 s.Fallback
If a custom
log_pam1is not reversible (its symmetrized form isn't symmetric) or has no positive stationary distribution,MutipleAlEnvfalls back to the originalEstimatePam. It stays available as the private_estimate_pam_referencefor tests.Tests
test/test_estimate_pam.pycompares the new code with the reference on 300 random alignments (10 to 1000 positions, all levels of divergence and gaps), on edge cases (identical sequences, saturated, gaps only,X,B/Z), and throughestimate_many_pam. It also checks the fallback with a perturbed non-reversible logPAM1.