Performance Measurement

Performance Testing Framework

MemCP includes an auto-calibrating performance test framework for regression detection and benchmarking.

Running Performance Tests

# Run performance tests (uses calibrated baselines)
PERF_TEST=1 make test

# Calibrate for your machine (run ~10 times to reach target)
PERF_TEST=1 PERF_CALIBRATE=1 make test

# Freeze row counts for bisecting regressions
PERF_TEST=1 PERF_NORECALIBRATE=1 make test

# Show query execution plans
PERF_TEST=1 PERF_EXPLAIN=1 make test

How Auto-Calibration Works

1. Target query time: 10-20 seconds
2. Each calibration run scales row counts by 30% up or down
3. Baselines are machine-specific, stored in .perf_baseline.json
4. Tests include 2 warmup runs before the measured run

Output Format

✅ Perf: COUNT (4.3s / 13s, 100,000,000 rows, 0.04µs/row, 25GB heap, 1522%/2400% CPU)
         │      │     │              │            │           └─ CPU utilization
         │      │     │              │            └─ Memory usage
         │      │     │              └─ Time per row
         │      │     └─ Calibrated row count
         │      └─ Threshold
         └─ Actual time

Bisecting Performance Regressions

# 1. Calibrate on known-good commit
git checkout <good-commit>
PERF_TEST=1 PERF_CALIBRATE=1 make test  # run 10x

# 2. Save baseline
cp .perf_baseline.json .perf_baseline_good.json

# 3. Run git bisect
git bisect start HEAD <good-commit>
git bisect run bash -c 'PERF_TEST=1 PERF_NORECALIBRATE=1 make test'

Environment Variables

Variable	Values	Description
PERF_TEST	0/1	Enable performance tests
PERF_CALIBRATE	0/1	Update baselines
PERF_NORECALIBRATE	0/1	Freeze row counts
PERF_EXPLAIN	0/1	Show query plans