divided up the text into summary, and contnt for NLP processing

[ertosns]

i divide up the text into two parts summary, and content, this can help in NLP processing specifically for summarizing transformers, for example the output of wikiextractor can be used to train wiki-summary https://github.com/ertosns/wiki-summary

[attardi]

Isn't

summary = [i for i in page]

the same as:

summary = page

[ertosns]

yes it is of course, but in case of numpy it will be a reference to it, i see in this case page is just a python default list, so you are absolutely right, perhaps i thought it was numpy! i will fix it now.

[ertosns]

also the output need to be pruned a bit, for example to add the option to fit certain criteria, for example some output is too long, or too short. i will work on that soon.