Skip to content

Fix Chinese money normalization for grouped and decimal values #491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
HungerDeng wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix Chinese money normalization for grouped and decimal values #491

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions nemo_text_processing/text_normalization/zh/taggers/money.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
import pynini
from pynini.lib import pynutil

from nemo_text_processing.text_normalization.zh.graph_utils import GraphFst
from nemo_text_processing.text_normalization.zh.graph_utils import NEMO_DIGIT, GraphFst
from nemo_text_processing.text_normalization.zh.utils import get_abs_path

suffix = pynini.union(
Expand Down Expand Up @@ -65,6 +65,15 @@ def __init__(self, cardinal: GraphFst, deterministic: bool = True, lm: bool = Fa

cardinal = cardinal.just_cardinals

# Accept conventional comma-separated numbers while keeping the cardinal
# grammar responsible for converting the separator-free digit sequence.
grouped_digits = (
(NEMO_DIGIT - "0")
+ pynini.closure(NEMO_DIGIT, 0, 2)
+ pynini.closure(pynutil.delete(",") + NEMO_DIGIT**3, 1)
)
cardinal = (cardinal | grouped_digits @ cardinal).optimize()

currency = pynini.string_file(get_abs_path("data/money/currency_major.tsv"))
currency_mandarin = pynini.string_file(get_abs_path("data/money/currency_mandarin.tsv"))
graph_digit = pynini.string_file(get_abs_path("data/number/digit.tsv"))
Expand Down Expand Up @@ -106,12 +115,10 @@ def __init__(self, cardinal: GraphFst, deterministic: bool = True, lm: bool = Fa
# larger money as decimals
graph_decimal = (
pynutil.insert('integer_part: \"')
+ (
pynini.closure(cardinal, 1)
+ pynutil.delete('.')
+ pynutil.insert('点')
+ pynini.closure((graph_digit | graph_zero), 1)
)
+ cardinal
+ pynutil.insert('\" fractional_part: \"')
+ pynutil.delete('.')
+ pynini.closure((graph_digit | graph_zero), 1)
+ pynutil.insert("\"")
)
graph_decimal_money = (
Expand Down
10 changes: 9 additions & 1 deletion tests/nemo_text_processing/zh/data_text_normalization/test_cases_money.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
$5~五美元
￥5~五人民币
$100~一百美元
$1,234~一千二百三十四美元
$1,234,567~一百二十三万四千五百六十七美元
€5~五欧元
£100~一百英镑
₩100~一百韩元
100美元~一百美元
1,234美元~一千二百三十四美元
500美元~五百美元
5元~五元
5毛~五毛
Expand All @@ -13,10 +16,15 @@ $100~一百美元
5块~五块
6毛~六毛
$1.5万~一点五万美元
$1234.0~一千二百三十四点零美元
$1,234.0~一千二百三十四点零美元
1,234.56美元~一千二百三十四点五六美元
3.5万韩元~三点五万韩元
1.5万美元~一点五万美元
我现在能拿出的现金是100000美元~我现在能拿出的现金是十万美元
一份煎饼卖5块一份~一份煎饼卖五块一份
每100美元能兑换700人民币左右~每一百美元能兑换七百人民币左右
您的银行账户余额为$500~您的银行账户余额为五百美元
洛杉矶的最低工资是每小时$15~洛杉矶的最低工资是每小时十五美元
洛杉矶的最低工资是每小时$15~洛杉矶的最低工资是每小时十五美元
价格是$1,234~价格是一千二百三十四美元
价格是$1234.0~价格是一千二百三十四点零美元