How much do language models memorize?
Paper • 2505.24832 • Published • 4
vocab_data int64 | bos int64 | model_vocab int64 | seq_len int64 | n_seq int64 | n_data_tokens int64 | data_seed int64 | sampling string | bits_per_token float64 | dataset_bits float64 |
|---|---|---|---|---|---|---|---|---|---|
2,048 | 2,048 | 2,049 | 64 | 5,000 | 320,000 | 100 | uniform iid over [0,2048) | 11 | 3,520,000 |
2,048 | 2,048 | 2,049 | 64 | 1,000 | 64,000 | 100 | uniform iid over [0,2048) | 11 | 704,000 |
2,048 | 2,048 | 2,049 | 64 | 2,500 | 160,000 | 100 | uniform iid over [0,2048) | 11 | 1,760,000 |
2,048 | 2,048 | 2,049 | 64 | 19,000 | 1,216,000 | 100 | uniform iid over [0,2048) | 11 | 13,376,000 |
2,048 | 2,048 | 2,049 | 64 | 4,000 | 256,000 | 100 | uniform iid over [0,2048) | 11 | 2,816,000 |
2,048 | 2,048 | 2,049 | 64 | 10,000 | 640,000 | 100 | uniform iid over [0,2048) | 11 | 7,040,000 |
2,048 | 2,048 | 2,049 | 64 | 10,000 | 640,000 | 100 | uniform iid over [0,2048) | 11 | 7,040,000 |
2,048 | 2,048 | 2,049 | 64 | 10,000 | 640,000 | 101 | uniform iid over [0,2048) | 11 | 7,040,000 |
2,048 | 2,048 | 2,049 | 64 | 90,000 | 5,760,000 | 100 | uniform iid over [0,2048) | 11 | 63,360,000 |
2,048 | 2,048 | 2,049 | 64 | 18,000 | 1,152,000 | 100 | uniform iid over [0,2048) | 11 | 12,672,000 |
2,048 | 2,048 | 2,049 | 64 | 44,000 | 2,816,000 | 100 | uniform iid over [0,2048) | 11 | 30,976,000 |
YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Tiny memorization synthetic datasets
Uniformly-random token datasets replicating the methodology of "How much do language models memorize?" (arXiv:2505.24832).
Each variant /<tag>/dataset_tokens.npy is an (N, 65) int64 array: a BOS token
(id 2048) followed by 64 uniform iid tokens in [0, 2048). The exact entropy is
H = N * 64 * log2(2048) = N * 704 bits (see each generation_params.json).
Tokens are sampled with numpy.random.default_rng(seed); the seed is recorded per variant.
- Downloads last month
- -