Methodology

Role quality and portfolio modes

The detailed ranking table exposes role-quality signals and cost-per-quality values. The eight recommendation cards are generated separately as constrained portfolios: Quality, Balanced and Budget.

For Balanced and Budget, cost outliers are identified first from the full latest-generation scored universe. After those models are removed, the best eligible Intelligence Index is recomputed; the requested 0.85/0.80 floor is then applied to that filtered baseline. If the resulting pool cannot support a feasible portfolio, the floor is lowered in 0.005 steps. Quality uses the unfiltered Intelligence baseline. Quality requires at least four families; Balanced and Budget require at least five. Every mode allows at most two seats per family.

There are no mandatory models, manual Fable blacklist, or per-role quality floor. Code Reviewer must use a different model family from both implementation roles. Implementer Heavy must differ from Implementer and exceed its Implementer coding-quality score (same role-specific benchmark metric for both models) by at least 2.0 points in Quality, Balanced and Budget.

Security Reviewer

Security Reviewer = Artificial Analysis Cyber Index v1, directly. The previous SciCode//HLE/Omniscience/LCR proxy has been removed. If a model has no direct Cyber Index score, it is simply ineligible for this one role.

Balanced and Budget share the same mean + 2σ plan-adjusted cost-outlier cutoff. Balanced combines cohort-relative quality and affordability percentiles (75% / 25%), then may trade up to 1% of the optimum to place expensive models in planning/review and cheaper models in execution. Quality and Budget use that placement rule only as a tie-break; Budget maximizes role-specific 40% quality / 60% affordability scores, using cost as a tie-break.

CommandCode credit burn and normalized reference workload

When Artificial Analysis publishes a complete task-token profile for a model, that observed workload is repriced with the model’s current CommandCode rates. If the profile is unavailable, the benchmark falls back to a versioned normalized reference workload. All token prices, discounts, billing categories and Max allowances come exclusively from CommandCode.

If CommandCode exposes no separate cache rate, normal input price is used. Active CommandCode promotions are retained.

CAI*: current observation, calibrated history, then estimator

If the current Coding Agent Index version has not re-evaluated a family, a prior observed CAI may be mapped onto the current scale with a validated overlap-derived degradation factor. Only families with neither current nor calibratable historical CAI use the estimator:

Ridge (λ=1) uses SciCode, HLE, LCR, GDPval, AA-Omniscience Index (normalized to 0–100) and log(Output Tokens/Task). 5NN uses SciCode, HLE and LCR, standardized across the observed CAI training set, with Euclidean distance and inverse-distance weighting over the five nearest models.

The estimator is trained only on unique model families with current-version observed AA Coding Agent results. Calibrated historical values never train the estimator, and commercial CommandCode variants never count as extra training examples.

SciCode fallback: current observation first; otherwise a last observed value no older than 14 days may be used only after overlap calibration against current observations passes MAE/max-error guardrails. The calibration factor is capped at 1. Ridge is a third-line fallback and is usable only when its original leave-one-out guardrails pass; otherwise the family is source-incomplete.

Coverage and failure rules

Universal weighted-role components, CommandCode Cost per Task and final CAI* must cover the scored model-family universe according to the published coverage rules. Security Reviewer is role-specific and uses direct Artificial Analysis Cyber Index coverage instead. Stealth/unresolved rows remain visible as UNSCORED. Validation continues to fail closed when estimator/ranking guardrails are breached.

Raw model mappings, current/historical-calibrated/estimated CAI*, SciCode fallback provenance and calibration diagnostics, Ridge/5NN components, role scores, task costs and final Ranking Values are all preserved in the public JSON snapshot.