PIF-Bench v2

In April we published PIF-Bench, a benchmark of AI donor research, and Rōmy came out on top with a score of 94.1. We have since rebuilt it from the ground up, because the first version could not carry the weight we were putting on it. This is the new version, its method, and its results — including the parts that do not flatter us.

Why we rebuilt it

PIF-Bench v1 scored each report from 0 to 100 on seven weighted dimensions, in one pass by one AI model. Read closely, that design had problems:

• Nothing behind the numbers could be checked. A "Factual Precision of 94" was a holistic judgment, not a count of claims anyone could audit.
• It rewarded our own format. "Structural Completeness" counted report sections; Rōmy's 16-section dossier scored 100 by construction.
• It graded things that cannot be known. Capacity was scored against "verified actuals" — but no one knows a living donor's true giving capacity.
• The answer key came after the answers. Some facts used to score recall were found by reading the systems' reports.
• The judge shared a family with one contestant, and the benchmark was built by the team behind another.

A benchmark that a fundraiser should trust has to be reproducible, specific, and built so its makers cannot tilt it. So we started over.

How v2 works

v2 follows the design of Harvey's Legal Agent Benchmark, the open benchmark for AI on legal work, adapted to the development office.

Real work, two tracks. The packet track has 20 tasks across prospect research, foundations and grants, gift compliance, major gifts, donor data, and finance and governance: verify a wealth-screening match where the richest hit is a namesake; brief a gift officer whose donor died three weeks ago; receipt a gala batch; value a year-end stock gift; read a funder's 990-PF; run the Schedule A public support test. Each task comes with a packet of synthetic records and the organization's own policies, so the right answer is known by construction. The live track asks for donor research on five real philanthropists — the prompts from v1 — and is graded on facts true on the day the reports were written, each with a primary source.

Pass/fail checks, written first. Every task has a rubric of 10 to 17 checks, each with explicit pass and fail conditions that name the exact figure, person or rule — "PASS if the gift is valued at $25,068.00, the mean of the day's high and low; FAIL if it uses the closing price ($24,820.00) or the sale proceeds ($25,215.00)." All 331 checks were written before any system ran. No weights, no 0–100 scores.

The headline is all-pass. A task counts only if every check passed — the way a supervisor reads a memo before it goes to a donor. We also report the share of checks met, and how often a system tripped a red line: soliciting someone who has died, merging two people, overstating a tax deduction, repeating a confidential gift.

The judges are tested too. Every task ships a correct deliverable and a flawed one with planted errors. A judge is trusted only as far as it catches those errors and passes the correct work.

Graded blind. Each report was graded one check at a time by a judge that never saw which system wrote it: system names and per-product citation tags were removed, no judge saw two answers to the same task, and the planted-error deliverables were mixed in unmarked. On the correct and flawed deliverables mixed into its pool, that judge returned all 662 verdicts right, and an independent second judging agreed with it on 99.6% of the verdicts both made.

Results

Real prospects: four products, April 2026

We re-graded the 16 reports Rōmy, Claude.ai, ChatGPT and Gemini wrote for v1 on Reed Hastings, Agnes Gund, Laurene Powell Jobs and Robert F. Smith.

No report met every check for any prospect. Every system contradicted the verified record somewhere: Netflix "co-founded in 1998" (it was 1997), a net worth of "$21–24 billion" (about $13.9 billion that day), Monumental Sports listed as a current holding after its December 2025 sale, The Atlantic placed at its former Watergate address, Reed Hastings's stake given as 4.8% of Netflix (it is under 1%). Rōmy made two of those errors.

Two caveats. Four prospects is a small sample, and the systems' ranges overlap. And the Claude.ai reports mention Rōmy by name although the prompts never did, so that run appears to have had Rōmy context in its project; it is not a clean product run.

Synthetic packets: six models, October 2026

The three leaders each missed a single check out of 259, so this set of tasks cannot separate them — our next tasks need to be harder. It does separate them from the rest, whose misses are the kind that reach donors: a grant budget with the wrong indirect-cost cap, a memo on a restricted endowment that never mentions asking the donor, a solicitation brief that repeats another family's anonymous gift.

What this means

The headline from v1 — that one system was "94% accurate" — does not survive a stricter test. The best product met 83% of the checks on real prospects, and none produced a report a careful researcher could send without corrections. That is not a reason to avoid AI in prospect research; it is the reason a researcher should verify what AI produces before it reaches a gift officer, and the reason we keep building Rōmy to show its sources.

Check our work

Everything is public: every task and rubric, every system's deliverable as written, every verdict with its reasoning, and the code that turns verdicts into scores. Anyone can re-grade the results with a different judge — including judges from three model families at once — with one command.

• The benchmark: github.com/getromy-app/pif-bench
• The method, and everything we changed from v1: METHODOLOGY.md
• The original April write-up: archived on GitHub

If you run prospect research and want to contribute a task from your own work, the contributing guide shows how.