Shannon Entropy LLAMA 2048

Entropy

Cross-corpus Shannon entropy comparison using LLAMA tokenization at a 2048-token window size.

Shannon Entropy WP 2048

Entropy

Cross-corpus Shannon entropy comparison using WORDPIECE tokenization at a 2048-token window size.

Shannon Entropy O200K 2048

Entropy

Cross-corpus Shannon entropy comparison using O200K tokenization at a 2048-token window size.

Shannon Entropy CL100K 2048

Entropy

Cross-corpus Shannon entropy comparison using CL100K tokenization at a 2048-token window size.

Shannon Entropy GPT-2 2048

Entropy

Cross-corpus Shannon entropy comparison using GPT-2 tokenization at a 2048-token window size.

CHARTS