Mahak aggregates results from two sources: community voting and rubric criteria. Here’s how each works.

Community Voting

The community votes on each output with an upvote or downvote. Duplicate votes from the same user on the same output are not counted — one vote per user per output.

Rubric Criteria

Each prompt has specific rubric criteria. The community evaluates each output against these criteria on a pass/fail basis. Rubric results are aggregated to calculate the compliance rate for each criterion.

Wilson Score

Mahak does not rely on a simple arithmetic mean — it biases toward low-vote outputs. Instead, we use the Wilson score lower bound at 95% confidence:

An output with 9 upvotes out of 10 gets a lower score than one with 90 out of 100, despite the same ratio.
This ensures that outputs with more votes get higher confidence.
Low-vote outputs are classified as “tentative” until they reach the confidence threshold.

Elo System for the Arena

The matrix shows the model’s recorded global arena rating. Until the model has played a recorded match, its rating is shown as “Unavailable”. Filtering the matrix by domain does not turn this global rating into a domain-specific rating.

In the arena, models are compared in double-blind pairwise rounds. After each round:

Both models’ Elo ratings are updated based on the outcome.
K-factor = 32 for new models (under 30 matches) to accelerate adaptation.
K-factor = 16 for calibrated models (30+ matches) for rating stability.
Expected probability is calculated via Bradley-Terry: 1 / (1 + 10^((ratingB - ratingA) / 400)).
The winner gains points and the loser loses the same amount — zero-sum.

Sample Quality Classification

The model summary is the mean Wilson lower bound across qualifying cells in the selected scope, expressed on a 0-100 scale. Empty cells and cells below that threshold are excluded from this mean. If none qualify, the summary is unavailable rather than zero. A measured zero remains a valid score.

A model summary is provisional unless every prompt in the selected scope has a calibrated cell for that model. Missing coverage therefore prevents the summary from being labelled calibrated. The existing tie-break rules apply to scored models; models without a score appear afterward.

Distinct raters counts people across a cell’s published outputs: two people rating three outputs produce six votes but only two distinct raters. This count does not itself establish statistical independence or change the current vote-count thresholds below.

Eligibility Policy (eligibility-1)

Cell eligibility is decided by one versioned, explainable policy — the same function powers live results, cached results, model summaries, exports and the progress labels. Every decision carries machine-readable reasons, and missing coverage is reported as missing, never as a low score.

A cell becomes provisional (rankable) with at least five votes from at least two independent raters (distinct vote user ids). A cell becomes calibrated with at least thirty votes, at least three independent raters, and at least two distinct contributors. Thirty votes by one person never become independent evidence, and unpublished outputs never count.

Every cell response carries eligible and eligibility_reasons — codes like independent_raters_below_provisional_minimum or votes_below_calibrated_threshold — plus the policy version and population definition, so any snapshot can be replayed and re-checked under the exact policy that produced it.

Uncertainty and Overlap Groups (uncertainty-1)

Model summaries carry an explicit evidence-strength block: the estimate, its 95% interval, the number of eligible cells and total votes, or an explicit “unavailable” when fewer than two eligible cells exist. The interval propagates each cell’s Wilson interval through the summary mean, including disagreement between prompts — closed-form and deterministic, so no resampling ever runs in a public request.

Models whose intervals overlap form connected components listed in indistinguishable_groups. This is a separation description only: connected overlap does not prove that models are equivalent in capability, touching endpoints count as overlap, and separated intervals are not a formal significance test. The matrix header shows the interval as text (with a screen-reader note that overlapping intervals are not a significance test and do not imply equal capability), so evidence strength never relies on color alone. Method reference: Chatbot Arena (ICML 2024) for cluster-aware resampling methodology — a direction for future revisions, not the current estimator.

Each result is classified by sample size:

0–4 votes (awaiting evaluation): shown in the matrix with a progress indicator inviting contributors to calibrate it.
Provisional: 5–29 votes — computed as the Wilson lower bound at 95% confidence.
Calibrated: 30 votes or more — a stable, fully calibrated result.

How Scores Are Calculated