Can a model execute an ABS waterfall to the cent?
WaterfallBench tests whether a model can read a securitization's legal documents and compute its monthly payments to investors exactly as the trustee did, to the cent. The answer key is what the trustee actually paid, as filed with the SEC.
Loading results…
The task
What a waterfall is
An asset-backed security turns thousands of car loans or leases into bonds, sold in classes from senior (A) to junior (B, C). Every month the borrowers' payments are collected and paid out in a fixed order of priority, the waterfall:
- 1Servicing feeto the servicer that collects the loans
- 2Interestsenior classes first
- Class A notesA-1, A-2, A-3, A-4 share pro rata
- Class B
- Class C
- 3Principalone class at a time, until each is paid off
- A-1→A-2→A-3→A-4→B→C
- 4Reserve accounttopped up to its required level
- 5Residualwhatever is left, to the equity holder
Each step is paid only from the cash left after the steps above it. Simplified: each deal's agreements set the exact order, amounts and targets.
Why it matters
Securitization funds a large share of US car loans and leases: the 20 deals in this benchmark alone issued $23.5 billion of notes. The waterfall decides who gets paid and in what order, so an error moves cash between senior and junior investors, and because each month starts from the last month's balances, it carries forward. Today this is done with deal-specific cash-flow models built by hand from the documents. WaterfallBench asks whether a model can read the documents and get it right on its own.
What the model gets
A deal's first distribution date. Opening balances come from the prospectus, so there is no prior report to copy from. All documents are EDGAR HTML as filed:
| Input | Filed as | What the waterfall needs from it |
|---|---|---|
| Final prospectus | 424(b)(5) | Note classes, coupons, day counts, the priority of payments |
| Closing agreements | 8-K exhibits | Indenture and servicing agreements: the fee, reserve and overcollateralization definitions |
| Servicer report | 10-D Ex. 99.1 | That month's collections, pool balances, rates and dates, with every waterfall result removed |
It must return the servicing fee, each class's interest, principal and ending balance, the reserve balance and the residual, in integer cents. Here is the report from a real task as the model sees it:
| Summary | Coupon | Initial balance | Beginning | Ending |
|---|---|---|---|---|
| Total Portfolio | 1,447,527,618.00 | 1,447,527,618.00 | 1,428,611,202.92 | |
| Class A-1 Notes | 3.949000% | 127,200,000.00 | 127,200,000.00 | REMOVED |
| Class A-2a Notes | 4.440000% | 414,000,000.00 | 414,000,000.00 | REMOVED |
| Class A-2b Notes | 3.978070% | 80,000,000.00 | 80,000,000.00 | REMOVED |
| Class A-3 Notes | 4.690000% | 494,000,000.00 | 494,000,000.00 | REMOVED |
| Class A-4 Notes | 4.780000% | 84,800,000.00 | 84,800,000.00 | REMOVED |
| Class B Notes | 4.870000% | 49,940,000.00 | 49,940,000.00 | REMOVED |
| Class C Notes | 5.080000% | 39,800,000.00 | 39,800,000.00 | REMOVED |
| III. Distributions | ||||
| Total Collections | 33,734,591.74 | |||
| 2. Reimbursement of Payment Advance | 0.00 | |||
| 4. Servicing Fee | REMOVED | |||
| 5. Class A-1 Notes Monthly Interest | REMOVED | |||
| Class A-1 Interest Carryover Shortfall | 0.00 | |||
Excerpt of fw-nissanlease-2026-b-2026-07. The same packet holds the 6.0 MB prospectus and 9 closing agreements.
Dataset
Tokens: mean prompt per task, Claude tokenizer, as sent by the harness. No task needed truncation.
Release checks
Every task passed these checks: the report's classes match the prospectus; each class's opening balance minus principal paid equals its ending balance, to the cent; class payments add up to the report's totals; no graded value is left in the packet; and the true answer grades 100% while an all-zeros answer resolves nothing.
Methodology
Harness
One API call per task: a fixed system prompt, the documents and the task. No tools, no code execution, no second turn, so the comparison is the model alone. EDGAR HTML is converted to text with table rows kept as cell | cell, then packed (report, prospectus, agreements) under a 1M-token cap; no task needed truncation. Reasoning effort is high for every model.
Evaluation metrics
Every task uses the same answer schema, given in the prompt. The model's reply is parsed, its classes are matched to the deal's by name, and every field is compared with the filed value, exactly.
{
"type": "object", "additionalProperties": false,
"required": ["servicing_fee_paid", "reserve_ending_balance",
"released_to_residual", "classes"],
"properties": {
"servicing_fee_paid": {"type": "integer", "description": "US cents"},
"reserve_ending_balance": {"type": "integer", "description": "US cents"},
"released_to_residual": {"type": "integer", "description": "US cents"},
"classes": {"type": "array", "minItems": 1,
"description": "one entry per class of notes the trust issued",
"items": {"type": "object",
"required": ["class", "interest_paid",
"principal_paid", "ending_balance"],
"properties": {
"class": {"type": "string",
"description": "the class name as the prospectus gives it, e.g. A-1"},
"interest_paid": {"type": "integer", "description": "US cents"},
"principal_paid": {"type": "integer", "description": "US cents"},
"ending_balance": {"type": "integer", "description": "US cents"}}}}}
}
{
"servicing_fee_paid": ✓,
"reserve_ending_balance": ✓,
"released_to_residual": ✗,
"classes": [
{"class": "A-1", "interest_paid": ✓,
"principal_paid": ✓, "ending_balance": ✓},
{"class": "A-2a", "interest_paid": ✓,
"principal_paid": ✓, "ending_balance": ✓},
{"class": "A-2b", "interest_paid": ✓,
"principal_paid": ✓, "ending_balance": ✓},
{"class": "A-3", "interest_paid": ✗,
"principal_paid": ✓, "ending_balance": ✓},
{"class": "A-4", "interest_paid": ✓,
"principal_paid": ✓, "ending_balance": ✓},
{"class": "B", "interest_paid": ✗,
"principal_paid": ✓, "ending_balance": ✓},
{"class": "C", "interest_paid": ✓,
"principal_paid": ✓, "ending_balance": ✓}
]
}
Values hidden: answer keys stay private. ✓ exact to the cent · ✗ wrong · ✓ trivial field (zero, or an opening balance left untouched).
| Metric | Definition | This output |
|---|---|---|
| Fully resolved primary | 1 if every graded field is exact to the cent, else 0 | 0 |
| Field accuracy | Share of non-trivial fields exactly right. Trivial fields are left out so untouched balances don't inflate the score | 9 / 12 = 75% |
| All fields | Share of all graded fields exactly right (the task-by-task grid) | 21 / 24 = 88% |
Leaderboard scores average these over the 20 tasks; ± is one standard error. Grading is deterministic, with no LLM judge. Cost uses each provider's list price for the tokens it reported.
Not graded. Auditing the runs turned up three places where the documents cannot determine a value. John Deere's first fee needs a pool balance no document prints. CarMax 2025-2's report departs from its own agreement's overcollateralization target. World Omni's agreement prorates the first fee without a day count. Those fields are left out of scoring; the filed values are never changed.
valkyrie run start --benchmark waterfallbench --dataset full-html \ --agent waterfallbench-direct --model anthropic/claude-haiku-5-5 \ -k effort high -s ANTHROPIC_API_KEY anthropic-api-key --concurrency 5
Results
Field accuracy vs task completion
Task by task
Green means every graded field is exact to the cent. Otherwise the cell shows the share of graded fields the model got exactly right.