Typecheck Regression

Type-check performance regression gate

scripts/check_typecheck_regression.py gates CI on whether axonc type-checking has slowed down per std module. This replaced the previous single-sample ratio-only check that produced a series of flaky P2/P3 pipeline-failure tickets (#99-#124) on shared autoscaler runners.

Algorithm

For each module in std/:

1. Take N=3 samples (configurable via N_SAMPLES env var).
2. Compute the median of the samples (robust to a single spike).
3. Compare against the frozen baseline:
   • Regression iff ratio > 1.5 AND median - baseline > 30 ms.
   • Improvement iff ratio < 1/1.5 AND baseline - median > 30 ms.

The dual threshold prevents both kinds of false positives:

Scenario	Ratio-only (old)	Abs-only	Dual (new)
Genuine 2× regression (+50ms)	✓ flags	✓ flags	✓ flags
Spike: 4× but only 2ms abs	✗ flags	✓ ignores	✓ ignores
Slow CI: 1.1× but 50ms abs	✓ ignores	✓ flags	✓ ignores
Genuine improvement	(didn’t detect)	(didn’t detect)	✓ flags

CI integration

performance:type-check-time:
  stage: test
  image: gcc:13
  needs:
    - project: labs/axon/axon-lang
      job: build:gcc-release
      ref: main
      artifacts: true
  before_script:
    - chmod +x build/axonc
    - chmod +x scripts/bench_typecheck.sh
    - chmod +x scripts/check_typecheck_regression.py
  script:
    - AXONC=build/axonc STDDIR=std N_SAMPLES=3 \
        scripts/bench_typecheck.sh > /tmp/samples.tsv
    - python3 scripts/check_typecheck_regression.py /tmp/samples.tsv \
        --baseline metrics/type_check_baseline.json \
        --output metrics/type_check_ms.json

Baseline

metrics/type_check_baseline.json holds median ms-per-module values. Values are seeded from the first CI run after this MR merges and updated over time by running locally with --update-baseline:

python3 scripts/check_typecheck_regression.py /tmp/samples.tsv \
    --baseline metrics/type_check_baseline.json \
    --update-baseline

Why the previous baseline caused false positives

The pre-existing baseline (e.g. string: 20ms) was tuned to a developer’s local Mac M1. On shared autoscaler runners, string.axs measured 94ms consistently — a 4.7× ratio that fired the gate on every push even when no real regression had occurred.

The new baseline (string: 100ms, see metrics/type_check_baseline.json) reflects observed CI behaviour with headroom. Future changes that bring type-check time down should regenerate the baseline to lock in the improvement.