... the ultimate solution for everything: frequency lists

the point: a unified approach based on frequency lists

1. 🔥 create plan: how to tackle each cleanness aspect using frequency lists:
   (1) fragment; (2) foreign; (3) spelling; (4) dedup

2. 🔥 implementation based on book-index/freqlists.py:
   (1) take script; (2) correct it (identify problem); (3) split to several scripts?; (4) solve all problems with it :)

command:

make sample FILENAME=? SAMPLESIZE=100000 RANDOMSEED=42

Are there too much foreign text in the corpus?

first attempt: based on the freq of most frequent words in certain languages

command:

scripts/investigate_foreign.sh FILENAME 42

results:

very few > rank=5000 > some > rank=1000 > quite much

Result is stable = RANDOMSEED=42 and RANDOMSEED=43 gives essentially the same.

Looking at concordance of the in MNSZ, it can be the case that even such many English text is not too much, because most hits are part of small English excerpts! Hm..

command:

scripts/investigate_spelling.sh FILENAME 42

results:

Result is stable = RANDOMSEED=42 and RANDOMSEED=43 gives essentially the same.

Idea: investigate MNSZ2 by subcorpora?

implementation: https://github.com/sassbalint/utils/blob/main/random_sampler.py (original source)

files (from arcj):

command: time make sample FILENAME=? SAMPLESIZE=?

results (average of 3 measures, in seconds):

• ➡️ time complexity: corpussize hardly matters (log?), samplesize below linear ➡️ extra efficient stuff! :)
• maybe slower first time = when loading corpus into memory
• there can be a limit if corpus is larger then the memory -- 🔥 to be tested
• con: stream not ok, whole file on disk is needed
• equally sized are the best for this algo -- solution can be: splitting long sentences into same sized chunks

OTKA pályázatbeli cleanness fogalom alapján, source

(1) é rtelmes = nincsenek széttöredezett szavak, szótöredékek, értelmetlen stringek
(2) m agyar = nincsenek benne idegen nyelvű többmondatos szakaszok;
(3) h elyesírás oké = ékezet, nagybetűsség, írásjelek okék (a normatív helyesírás nem követelmény);
(4) d eduplikált = nincsenek benne azonos többmondatos szakaszok.

(a boilerplate is megoldható dedup alapon?)

tok or split by space
sentsplit or split by ≈100 chars or ≈7 words
➡️ spl
➡️ base unit = "sentence"