SlayerLab/Slayer1B-1.07

Slayer1B 1.07 (training in progress)

Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. Slayer1B 1.07 continues pretraining of Slayer1B 1.06 from its final checkpoint (step 13848) on 8 891 005 707 training tokens (about 7.0 B new and 1.9 B replayed from the 1.06 data), at the same constant learning rate and global batch.

Status: training in progress. The planned end is step 22327. This card reports intermediate training-time evaluations up to step 19000 and will be replaced by the final card when training ends. No 1.07 weights are published in this repository yet. Intermediate training checkpoints (raw PyTorch training state ckpt_N.pt, not a ready-to-use model) are published file by file in SlayerLab/Slayer1B-1.07-checkpoints, with sha256 sums.

• Previous version: SlayerLab/Slayer1B-1.06 (step 13848, 14 520 680 448 tokens, one full pass over SlayerLab/slayer-pl-8x3b)
• Training data of 1.07: SlayerLab/slayer-1.07-pretrain
• Training curve and training-time evaluation: https://track.fabryka.ai/run/4973c068-8dd5-59d8-b6c4-671ca3beb925

Full-size chart

Training

• Start: Slayer1B 1.06 checkpoint ckpt_13848.pt (sha256 1fc7396ca1790805128244b2f827e02b9fe88e352534cd887d0c488f121d1e4d).
• Same model, tokenizer (v6, 32 768 tokens, sha256 0640d3bd…) and global batch as 1.06: 256 sequences × 4096 tokens = 1 048 576 tokens per step, constant learning rate 6e-4, no learning-rate decay, no context extension.
• Hardware: 4× NVIDIA RTX 4090 (gradient accumulation 32, so the global batch is unchanged from the 8-GPU runs), about 45 000 tokens/s. Trainer train_gpt_ref r24.2 (sha256 46f83b51…), unchanged since 1.05.
• Planned length: steps 13848 → 22327 (8 479 steps, one pass over the new data; each window is used once).
• Training was interrupted on 4 November at step 17130 by an external stop of the process and resumed on 5 November from the saved checkpoint at step 17000. After the resume the loss at step 17010 was identical to the first run (2.3177), so the resumed run reproduces the original one. A second external stop on 5 November at step 19330 was resumed within two minutes from the checkpoint at step 19250; the loss at step 19260 was again identical (2.3105).

Training data of 1.07

8 891 005 707 tokens in 9 files, after removing documents that overlap evaluation sets (13-gram scan) and after masking personal data (e-mail addresses, phone numbers, national ID numbers) with the same procedure as in 1.06.

source	language, type	documents (built)	tokens (built)	tokens in training	license
HPLT (Polish web, parts not used in 1.06)	PL, general web	2 448 525	3 119 999 086	3 110 430 927	compilation CC0; texts used under the text-and-data-mining exception (Art. 4, Directive 2019/790)
Polish core texts, repeated from 1.06	PL, curated texts	3 165 531	999 785 312	982 781 646	per source, as in slayer-pl-8x3b
Polish legal texts, repeated from 1.06	PL, legal	261 590	1 004 151 322	911 514 295	per source, as in slayer-pl-8x3b
FineWeb-Edu (from minimal-en-corpus-5b)	EN, educational web	840 133	978 166 858	one file, 1 650 618 814 together	ODC-By
FineMath	EN, mathematics	152 623	237 458 786	(same file)	ODC-By
open-web-math	EN, mathematics from the web	86 621	225 412 736	(same file)	ODC-By
DCLM-baseline (subset)	EN, general web	177 946	244 996 577	(same file)	CC BY 4.0
FineWeb-Edu sample/10BT	EN, educational web	2 035 410	2 254 202 363	2 235 660 025	ODC-By

Documents in the published training files (after leak removal): Polish web 2 445 601, Polish core 3 151 339, Polish legal 258 328, English mix 1 249 258, FineWeb-Edu sample/10BT 2 029 345 (9 133 871 in total).

Share of training tokens: Polish web 35.0 %, Polish core 11.1 %, Polish legal 10.3 %, English 43.7 %. The Polish core and legal parts are a random (hash-ordered) subset of what 1.06 already saw; about 36 % of the 1.06 core tokens and 68 % of the 1.06 legal tokens are seen a second time, the rest of the data once. English documents whose exact normalized text was already in the 1.06 English data were excluded.

Training-time evaluation

Same automatic evaluator and the same zero-shot protocol as the 1.06 card (BLiMP, ARC-Easy, WikiText-2, MultiBLiMP-PL, ARC-Easy-PL eu20). 1.07 checkpoints are evaluated on CPU; the 1.06 checkpoint (step 13848) evaluated on the same CPU setup matches the 1.06 card values at card rounding. These are not the lm-evaluation-harness benchmark used in the Results table of 1.00–1.03.

metric	1.06 (13848)	13998	14000	15000	15730	16000	17000	18000	18590	19000
BLiMP (EN), accuracy %	76.12	76.29	76.33	77.94	77.32	77.13	77.61	77.88	77.46	77.74
ARC-Easy (EN), accuracy %	52.99	54.34	54.21	54.59	55.64	56.73	56.48	56.52	56.52	56.65
WikiText-2, byte perplexity ↓	2.2849	2.2618	2.2601	2.2321	2.2237	2.2205	2.2138	2.2042	2.2145	2.2029
WikiText-2, token perplexity ↓	30.03	28.80	28.71	27.28	26.85	26.70	26.37	25.90	26.40	25.84
MultiBLiMP (PL), accuracy %	98.41	98.66	98.59	98.62	98.62	98.81	98.87	98.84	98.62	98.53
ARC-Easy (PL, eu20), accuracy %	45.03	45.37	44.91	45.92	45.75	46.25	45.41	46.21	47.14	46.80

ARC-Easy has 2 376 questions per language, so differences below about one percentage point between neighbouring checkpoints are within noise. WikiText-2 byte perplexity at 18590 was 0.010 higher than at 18000; at 19000 it is the lowest so far (2.2029). Each point is a single measurement of a constant-learning-rate run. The training loss in the figure is measured on the 1.07 data mixture (44 % English), so it is not directly comparable with the training loss of 1.00–1.06.

The benchmark caveat of the 1.03 card applies to the 1.00–1.06 part of the training. For 1.07, all nine data files were scanned for 13-gram overlap with the test items of the evaluation suite before training, and documents with any hit were removed. Exact matches of short answers (below 13 words) were counted but not removed, and one evaluation set (EWoK, distributed as a password-protected archive) was not part of the scan. The Slayer 1.06 core includes the source passage corpora of PolQA and PoQuAD; passages with a 13-gram hit were removed and the rest remain in the replay.