Dataset Expansion and Training & Loss Formulation Hangul Pos Tokenizer - #887
Open
kahye wants to merge 2 commits into
Open
Dataset Expansion and Training & Loss Formulation Hangul Pos Tokenizer#887kahye wants to merge 2 commits into
kahye wants to merge 2 commits into
Conversation
…, and resilience experiment report
…ility evaluation suite - Expand Korean POS dataset in get_dataset.sh with OPUS-100 and KLUE task splits (DP, NER, MRC, NLI, RE, STS, YNAT) and update lane_metadata.json - Add --pos_loss_weight argument in train_args.py and handle POS loss weighting and milestone checkpoint saving in train.py - Add --mc_ckpt and --base_ckpt path override support to benchmarks/run_phonetic_slang_eval.py and benchmarks/run_vocab_tail_perplexity.py - Add 4-capability evaluation benchmark suite (benchmarks/run_four_capability_evals.py) covering KLUE-NER, KLUE-DP, noisy text resilience (NSMC/UnSmile), and rare vocabulary/OOV (KorMedMCQA) - Add evaluation runner demos/run_all_epoch_evals.sh, Option 1 sweep runner run_option1_sweep.py, and 10-epoch experiment runner run_opt1_10ep_experiment.py
kahye
force-pushed
the
pos-tokenizer-more-data
branch
from
August 11, 2026 19:19
8371131 to
842c30f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
• get_dataset.sh: Expanded corpus generation by integrating full OPUS-100 en-ko with all KLUE task splits (dp, ner, mrc, nli, re, sts,
ynat).
• lane_metadata.json: Updated total character count to 21,065,854.
• train_args.py: Added --pos_loss_weight CLI parameter.
• train.py: Supported custom POS loss weighting (wₚₒₛ) alongside structural loss down-weighting (
), and added milestone epoch checkpoint savings (iterations 3472, 5786, 8100, 11572).
• run_four_capability_evals.py: Comprehensive 4-capability benchmark evaluation covering:
1. Token-Level Extraction (KLUE-NER)
2. Syntactic Parsing (KLUE-DP)
3. Informal & Noisy Text Resilience (NSMC & UnSmile @ 80% noise)
4. Rare Vocabulary / OOV Generalization (KorMedMCQA)
• run_phonetic_slang_eval.py & run_vocab_tail_perplexity.py: Added --mc_ckpt and --base_ckpt explicit checkpoint path override flags.
• run_option1_sweep.py: Automated training sweep and 4-capability evaluation across loss weights
.
• run_opt1_10ep_experiment.py: 10-epoch training and evaluation progression runner.
• run_all_epoch_evals.sh: Script to run evaluations across 3, 5, and 10 epoch checkpoints.