#replication
2 experiments
EXP-011
Cross-Model Replication: Surprisal Typology Clusters by Family in Both Qwen and Gemma 4
Does the finding that surprisal curves cluster by language family replicate across different model architectures (Qwen2.5-7B dense vs Gemma 4 E2B MoE)?
- Family clustering replicates across architectures, but with important caveats. Gemma 4 shows 2.52x family ratio vs Qwen'…
- The within-family distance appears identical (0.0073) across both models — this is a rounding coincidence. Actual values…
#replication#surprisal#typology#multilingual
EXP-004
MI-Weighted BPE Merges: A Promising Result on Portuguese That Failed to Replicate Across 4 Languages and 2 Domains
Does weighting BPE merge decisions by mutual information between boundary bytes improve language modeling, and does the effect depend on language morphology or text domain?
- Only 1 of 7 direct comparisons shows improvement. MI-weighted BPE achieved a -2.90% BPB gain on the Portuguese Carolina …
- The morphological complexity hypothesis is falsified. Turkish — the most morphologically complex language tested, with p…
#negative-result#tokenization#bpe#mutual-information#cross-lingual