VibeRepair+

Summary

VibeRepair+ extends a specification-centric repair paradigm to Java and Python with a hybrid retriever (Reciprocal Rank Fusion over parallel dense and lexical retrievers), LLM-based distillation of fix guidance, and removal of an external tool dependency — cutting token consumption from 1,564 to roughly 150 tokens per query.

My part of the work was the retriever framework and the evaluation. The evaluation is the part I would most want a reviewer to read, because the headline result is that the protocol everyone was using could not distinguish the retrievers at all.

A framework where adding a retriever is one file

The retriever layer sits over a 97,402-example Java bug-fix corpus. Adding a retrieval strategy requires one Python file and zero changes to the serving endpoint, the evaluation harness, or the metrics. Reproducibility is enforced by a SHA-256 fingerprint over the corpus and a locked holdout split, so two runs that report different numbers cannot be silently reading different data.

Three retrievers are compared under it: a dense embedding baseline (ada, ~285 ms p50), lexical BM25-Okapi, and a hybrid that runs both in parallel and fuses by RRF rank summation with k = 60.

The evaluation was measuring a self-match

The incumbent protocol is leave-one-out over the corpus: hold out an example, query with it, check whether its own fix comes back. Under it, all three retrievers score a perfect recall@1 = 1.000 and MRR@10 = 1.000. That looks like a solved problem.

It is an artifact. After excluding the self-match, the retrievers agree on only 34.1% of top-1 results, with a top-10 Jaccard of 0.166 — three systems returning substantially different evidence, all scoring identically. LOO was measuring whether a document can find itself, which every retriever can do.

The replacement protocol:

• External queries. 521 real Defects4J bugs, none drawn from the corpus (533 minus a leak filter, an empty-delta case, and 11 lacking annotations).
• Operator-aware proxy ground truth. Unified diffs with Jaccard overlap over tokens, with operators preserved — recovering 170 of 171 bugs that a word-only tokenizer left with empty deltas because the fix was operator-only (== → !=). The top-5 corpus rows by Jaccard form the positive set.
• A two-stage leak check (embedding cosine > 0.95 and normalized Levenshtein > 0.95), which finds zero byte-overlap — the evaluation is genuinely external.
• A random-retrieval floor reported alongside everything: recall@10 ≈ 5×10^-4, so the dense baseline at 0.138 is a 276× improvement over chance and the task is meaningfully discriminative even where absolute numbers look low.

Results under the corrected protocol