TL;DR
On 23 shared calibration fixtures, Jev 1.14 found 15 of 22 labeled defects across three saved mapper attempts per fixture. Saved GPT-5.6 Luna found 21. On reducer inputs containing candidate findings, Jev matched 40 of 54 expected verdicts; Luna matched 44.
This compares October Jev responses with July Luna responses on saved artifacts and constructed controls. We tested the grading components separately. This experiment launched zero fresh run_compute jobs, and the Luna results are historical. Production grading is unchanged.
A correct SQL filter drew seven issue flags
The control asks for the latest closing price for every non-tech name. Our data contract uses nullable industry flags, so the requested population includes FALSE and NULL values. The control query uses technology IS NOT TRUE, which keeps both.
Jev flagged population, completeness, fidelity, support, statistics, contract and other on that correct query in all three attempts. Luna raised no issues on this fixture. These seven category flags describe allegations on one query; the hand-labeled control contains zero defects.
Across all ten clean controls, Jev proposed at least one issue in 23 of 30 attempts, compared with Luna’s 13. This frequency measures how often correct work attracted an allegation.
Mapper and reducer results
Show the table · 5 rows, 3 columns
Scroll sideways to compare all columns. Select a heading to sort.
| Mapper defects found in any of 3 attempts | 15 / 22 | 21 / 22 |
| Clean-control attempts with any issue | 23 / 30 | 13 / 30 |
| Reducer verdicts correct, nonempty inputs | 40 / 54 | 44 / 54 |
| Reducer defects retained in any of 3 entries | 20 / 22 | 20 / 22 |
| Reducer verdicts correct, including empty inputs | 52 / 69 | 56 / 69 |
No matching rows. Clear the filter to see all records.
Repeated attempts reuse the same 23 fixtures. They are not 69 independent cases. Higher is better except for clean-control allegations.
A mapper reads an artifact and proposes defects. A reducer receives candidate findings and evidence, then decides which findings to retain, how severe they are and which describe the same problem. We reviewed each role separately.
The shared mapper corpus contains 22 labeled defects across 13 defective fixtures, plus ten clean controls. We reviewed 220 distinct Jev claims and 204 distinct Luna claims against those labels and the located evidence. A repeated claim of the same defect counts once per fixture.
Jev identified 15 of 22 defects (68.2%) and Luna identified 21 (95.5%) at least once across three attempts. These percentages describe pooled coverage across attempts, not the probability that a single attempt finds a defect.
The reducers received the same frozen candidate lists and evidence for each entry. On the 54 entries with candidates, Jev matched 40 expected verdicts (74.1%) and Luna matched 44 (81.5%). Both retained 20 of 22 labeled defects at least once across the three repeated entries per fixture. Retaining a defect and choosing the expected final verdict are separate measures.
Why the two roles need separate tests
In research-continuity-resolved, the gold defect concerns the realization of training labels. Jev’s mapper returned no findings on any of its three attempts. The saved Luna mapper identified that defect.
The frozen reducer inputs for this fixture contained no candidate findings. Neither reducer was called on those three entries, so neither could repair the upstream omission. Across the corpus, 15 of 69 reducer entries had empty candidate lists: 12 clean entries and these three defective entries.
That is why the headline reducer comparison uses the 54 entries with candidates. Including the 15 empty entries produces 52 of 69 correct verdicts for Jev and 56 of 69 for Luna. The downloads preserve both views.
What ran, and how we scored it
Read the full test method
The Jev 1.14 experiment ran on October 15, 2026. The Luna comparison uses archived openai/gpt-5.6-luna responses from July 30. We recovered 345 complete Luna mapper calls and 54 reducer calls and joined them to the saved fixture inputs. Reviewing these records made zero new inference calls.
The corrected paired comparison uses the 23 shared main-corpus fixtures. These include constructed controls and saved research evidence. The broader Jev exploration also included nine code and nine vision fixtures; their old keyword recall percentages are excluded from these results.
The mapper configurations differ. Jev used its Decisions API adapter with category questions and selected evidence. The July Luna configuration used three code-reader calls, six log-reader calls and six output-reader calls per fixture across the three attempts. Policies also differ between the July and October runs, so this measures the recorded configurations rather than isolating the model alone.
The archived mapper requests did not preserve their workflow attempt identifiers. We could join them to fixtures, so we report coverage pooled across all three attempts per fixture. We do not reconstruct a single-attempt recall percentage from call order.
Codex manually reviewed the observed claims, the gold defects and their evidence. A match requires an asserted defect at the relevant location. Category names, generic cautions, negated allegations and copied evidence alone earn no credit. The review records include reasons and are bound to the exact scoring inputs. This was an assistant review, without an independent external judging panel.
For reducer verdicts, we applied the same frozen-corpus rule to both saved sets: a retained artifact-owned blocking defect or a blocking source-verification hold fails the entry; major warnings alone pass. This matches the corpus’s warning policy. Today’s production pass rules also treat major artifact defects as failures, so the reported verdicts describe this benchmark’s rule.
The gold labels are not exhaustive. We distinguish known false allegations from claims outside those labels and do not report an overall precision score. The corrected finding scores replace substring matching, which had credited a generic mention of selection as a multiple-comparisons finding.
We retained the raw calls and private grading policies locally. Public downloads contain the reviewed metrics, per-entry verdicts and the selected SQL example.
How good was Jev 1.14 in this test?
Saved Luna found six more labeled mapper defects and raised issues on ten fewer clean-control attempts. Its reducer matched four more expected verdicts on the 54 entries containing candidates. The two reducers tied on pooled labeled-defect retention.
These results favor the saved Luna configuration on discovery and verdict agreement. A comparison against our current run_compute grader still requires current configurations and representative completed jobs. Production grading is unchanged.
No comments yet.