how scores are computed
Length-weighted families. Each eval family (clean ppl, junk ppl, needle acc) is
collapsed across the 6 eval lengths with a weighted average, weight = L / 2048
(so 2048→1×, 4096→2× … 65536→32×). Longer-context behaviour dominates. → columns
clean_ppl_wavg, junk_ppl_wavg, clean_bpb_wavg,
junk_bpb_wavg, clean_bits_gpt2tok_wavg,
junk_bits_gpt2tok_wavg, needle_acc_wavg.
perf_index. Each family's length-weighted value is standardized (z-scored:
(x − mean) / std) across all done runs; the two perplexity z-scores are sign-flipped
so higher is always better. perf_index is the mean of the three z-scores — a field-relative
score centred at 0 (positive = above the average run).
eff_index. Efficiency-adjusted score. MFU is z-scored across done runs and
added to perf_index: eff_index = perf_index + 0.5 × z(MFU). Additive (not
multiplicative) so there's no absorbing zero — MFU always shifts the score, and a high-MFU run can
climb even from low perf. The 0.5 weight keeps performance dominant; MFU is a secondary adjustment.
group view. Picking a group by factor (e.g. attn_type) shows one row per value, with every metric averaged over all other axes, isolating that single factor.
Note: z-scores are relative to the runs completed so far, so absolute index values shift as more cells finish — rankings are stable, the numbers drift.