Skip to content

Feat/danish normalization replacement bundle#29

Open
agf-cmd wants to merge 5 commits into
gladiaio:mainfrom
agf-cmd:feat/danish-normalization-replacement-bundle
Open

Feat/danish normalization replacement bundle#29
agf-cmd wants to merge 5 commits into
gladiaio:mainfrom
agf-cmd:feat/danish-normalization-replacement-bundle

Conversation

@agf-cmd

@agf-cmd agf-cmd commented Jul 7, 2026

Copy link
Copy Markdown

What does this PR do?

Added a bundle of extra features for Danish based on native Danish knowledge and verified by ordnet.dk. Including:

  • adding ordinals and "tusinde" to number_normalization.py.
  • added a sentence_replacement.py
  • adding "dobbeltformer" (words with multiple accepted spellings) and abbreviations to replacement.py
  • adding new tests required

Type of change

  • New language
  • Edit existing language (fix a replacement, tweak config, …)
  • New normalization step
  • Edit existing step (bug fix, behaviour change)
  • New preset version
  • Bug fix (other)
  • Refactor / docs / CI

Checklist

Only fill in the section(s) that match your change — delete the rest.


Edit existing language

  • New/changed word substitutions go in replacements.py, not inline in operators.py
  • If you changed a config field that can be None: the step reading it still handles None gracefully
  • Unit tests updated or added
  • E2e CSV updated if the expected output changed

How was this tested?

uv run pytest tests/

Summary by CodeRabbit

  • New Features
    • Expanded Danish text normalization with more spelling and phrase variants.
    • Added support for Danish ordinals, including forms like første, anden, 1., and glued compounds like enogtyvende.
    • Improved handling of thousand forms so tusinde is normalized like tusind.
  • Bug Fixes
    • Better normalization of common abbreviations and multi-word phrases into standard Danish forms.

agf-cmd and others added 4 commits July 7, 2026 12:45
…d sentence replacements

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ntence replacements

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@agf-cmd, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 32 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 44e087df-2d87-4215-8db3-0381a55bba4d

📥 Commits

Reviewing files that changed from the base of the PR and between 9112d74 and 630aad6.

📒 Files selected for processing (1)
  • normalization/languages/danish/replacements.py
📝 Walkthrough

Walkthrough

Adds Danish ordinal number parsing (including glued compounds and cardinal+ordinal merging) and "tusinde" plural support to the number normalizer, wires new sentence replacements into the operators config, introduces DANISH_REPLACEMENTS and DANISH_SENTENCE_REPLACEMENTS dictionaries, and adds corresponding unit tests.

Changes

Danish normalization enhancements

Layer / File(s) Summary
Ordinal vocabulary and compound parsing
normalization/languages/danish/number_normalizer.py
Adds ordinal word-to-value lookup tables and a helper to parse glued "og" ordinal compounds (e.g., enogtyvende → 21.).
Ordinal integration into parsing flow
normalization/languages/danish/number_normalizer.py
Merges cardinal + trailing ordinal continuation into one token, and falls back to standalone ordinal detection when no cardinal is parsed.
Tusinde plural support
normalization/languages/danish/number_normalizer.py
Treats "tusinde" the same as "tusind" in the multiplier dictionary, sub-999 follow-on check, and continuation predicate.
Operators config wiring
normalization/languages/danish/operators.py
Imports and wires DANISH_SENTENCE_REPLACEMENTS into DANISH_CONFIG, and adds "tusinde" to number_words.
Word and sentence replacement dictionaries
normalization/languages/danish/replacements.py, normalization/languages/danish/sentence_replacements.py
Adds DANISH_REPLACEMENTS (colloquial/abbreviation word mappings) and DANISH_SENTENCE_REPLACEMENTS (phrase mappings) as new dictionaries.
Tests for ordinals, tusinde, and replacements
tests/unit/languages/danish_number_normalizer_test.py, tests/unit/languages/danish_operators_test.py
Adds parametrized tests validating ordinal parsing, tusinde equivalence, and word/sentence replacement mappings.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

  • gladiaio/normalization#25: Both PRs modify the same Danish number normalization/operator components, extending the foundational Danish support with ordinal/tusinde/sentence-replacement logic.

Suggested reviewers: karamouche

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and related to the Danish normalization updates, though it is broader than the specific feature set.
Description check ✅ Passed The PR description matches the template with purpose, type of change, checklist, and testing notes, though the summary is a bit broad.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@normalization/languages/danish/replacements.py`:
- Around line 51-52: The demonym replacement entry is using an uppercase key,
but the normalization flow lowercases text before applying word replacements, so
the key will never match. Update the replacement mapping in the Danish
replacements table so the `Ægypter` entry uses the lowercase form `ægypter` as
the lookup key while keeping the normalized value `egypter`, alongside the other
entries in the same replacements mapping.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1586f667-78d9-4487-9212-8fe428a0333c

📥 Commits

Reviewing files that changed from the base of the PR and between 824db67 and 9112d74.

⛔ Files ignored due to path filters (1)
  • tests/e2e/files/gladia-3/da.csv is excluded by !**/*.csv
📒 Files selected for processing (6)
  • normalization/languages/danish/number_normalizer.py
  • normalization/languages/danish/operators.py
  • normalization/languages/danish/replacements.py
  • normalization/languages/danish/sentence_replacements.py
  • tests/unit/languages/danish_number_normalizer_test.py
  • tests/unit/languages/danish_operators_test.py

Comment thread normalization/languages/danish/replacements.py Outdated
… lowercasing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@remilejeune2 remilejeune2 requested a review from karamouche July 9, 2026 14:41

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add some tests for ordinality, and for abbreviations replacements here?

@karamouche

Copy link
Copy Markdown
Collaborator

Thanks for your contribution @agf-cmd! Here are two things to do to fix the CI's issues :

  • Rename commit d160fa5 with a correct tag respecting commit lint conventions (can be feat: ... here)
  • Run uv run pre-commit install --install-hooks to install local pre-commit hook, which will apply/highlight the necessary lint on your code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants