Add scripts to preprocess WARP-pdf - #64

Open

hkiyomaru wants to merge 23 commits into

mainfrom

llm-jp-corpus-v4-warp-pdf

Commits on Jan 16, 2025

add conversion script
hkiyomaru
committed
fc92729
workaround
hkiyomaru
committed
89c367e
update readme
hkiyomaru
committed
ad30f44
tweak
hkiyomaru
committed
75f5a2f
[wip] concurrent processing
hkiyomaru
committed
0ef3271
tweak
hkiyomaru
committed
f33b28d
fix
hkiyomaru
committed
91afaea
only process span around newline
hkiyomaru
committed
bd1312e
lint
hkiyomaru
committed
46a8c8b

Commits on Jan 17, 2025

update dependencies
hkiyomaru
committed
b940da5
fix
hkiyomaru
committed
8a8ea03

Commits on Jan 18, 2025

keep order
hkiyomaru
committed
00a8bee
use cpu count as default parallelism
hkiyomaru
committed
f53a330

Commits on Jan 20, 2025

add overwrite option
hkiyomaru
committed
1ff7b3b
fmt
hkiyomaru
committed
9f9b787
udpate
hkiyomaru
committed
fd49bd2
fix
hkiyomaru
committed
3064667

Commits on Jan 23, 2025

fix
hkiyomaru
committed
07d5953

Commits on Jan 30, 2025

update
hkiyomaru
committed
1dd1275
remove docs wo meta
hkiyomaru
committed
a8d5216
add launch.py
hkiyomaru
committed
9cea63f
fmt
hkiyomaru
committed
e2d476b
fix
hkiyomaru
committed
99f1f7c