Chess Engine Evaluation: Confidence Intervals and Error Rates

For elite chess engines like Stockfish 17 and Torch, acceptable confidence intervals are remarkably tight due to their near-perfect play. Typical standards include ±10-20 centipawns for evaluation accuracy and 95-99% move matching with optimal play, with error rates below 1-2% for critical positions.

Understanding Centipawns

Chess engines measure positional advantage in centipawns (cp), where 100 centipawns = 1 pawn advantage. Modern engines like Stockfish 17 typically evaluate positions with precision measured in single-digit centipawn differences.

[-300cp = Lost] ← [-100cp = Bad] ← [0cp = Equal] → [+100cp = Good] → [+300cp = Winning]

Evaluation Accuracy Standards

Position Evaluation Confidence

Standard Position Evaluation: ±10-20 centipawns

For most positions, engines should maintain evaluation stability within this range across different runs and time controls.

Critical Positions: ±5-10 centipawns

In tactical or endgame positions where precise evaluation matters most, tighter tolerances are expected.

Tablebase Positions: Exact evaluation required

For positions with 7 pieces or fewer, Syzygy or Lomonosov tablebases provide perfect play, so engine evaluations should match exactly.

Move Selection Accuracy

Move Matching Rates

Scenario	Acceptable Match Rate	Error Rate
Forced tactical sequences	99-100%	< 1%
Critical winning moves	98-99%	1-2%
Positional decisions	90-95%	5-10%
Opening theory moves	95-98%	2-5%
Endgame technique	97-99%	1-3%

Time Control and Resource Dependencies

Performance vs. Thinking Time

Ultra-fast (bullet): 80-90% of optimal play

At 1-2 minutes/game, engines make more practical decisions rather than finding absolute best moves.

Rapid (10-30 minutes): 95-98% of optimal play

Standard testing conditions where engines approach their full strength.

Correspondence (days/move): 99%+ of optimal play

With unlimited analysis time and tablebases, error rates should approach zero.

Testing and Validation Standards

Engine Testing Framework

Statistical Significance in Testing

When comparing engine versions, chess engine testers typically require:

- 95-99% confidence in Elo difference measurements

- ±2-5 Elo points margin of error for rating estimates

- Thousands of games to establish reliable ratings (typically 10,000+ games for small differences)

False Positive Rates

Acceptable false positive rates for claiming engine improvement are typically < 5%, with more conservative projects requiring < 1%.

Special Considerations for Neural Network Engages

Engines like Torch (and Leela Chess Zero) have different error characteristics:

Evaluation Consistency: NNUE and neural network engines can show more evaluation volatility but often find better strategic plans

Monte Carlo Tree Search Effects: Probability distributions over moves create natural confidence intervals

Training Artifacts: Neural networks may have systematic blind spots that require specialized testing

Practical Error Rate Benchmarks

Error Type	Stockfish 17 Standard	Tournament Acceptable
Blunder rate (>100cp error)	< 0.1% of moves	< 0.5% of moves
Mistake rate (>50cp error)	< 0.5% of moves	< 2% of moves
Inaccuracy rate (>20cp error)	< 2% of moves	< 5% of moves
Tablebase mismatch	< 0.01%	< 0.1%
Opening book errors	< 0.5% of book moves	< 2% of book moves

Summary: The Gold Standard of Chess Accuracy

For elite engines like Stockfish 17 and Torch operating at high time controls, the community expects near-perfect play:

Evaluation Precision: Centipawn-level accuracy (±10-20cp) in most positions, with essentially perfect evaluation in tablebase positions

Move Selection: 95-99% agreement with optimal play across different position types, with the highest standards for forced sequences and winning positions

Statistical Reliability: 95-99% confidence in testing results, requiring thousands of games to establish small Elo improvements

Practical Performance: Blunder rates below 0.1% and consistent superiority over all human players (3500+ Elo performance)

The remarkable aspect of modern chess engines is that they operate at accuracy levels that would be considered extraordinary in most computational domains, demonstrating that for well-defined problems like chess, computational methods can approach theoretical perfection.