WaterfallBench
Benchmarks / Finance

WaterfallBench

Updated 11/15/2026Version 1.0Code & tasks ↗
WaterfallBench v1.0First-distribution waterfalls
vs cost / task
Leaderboard20 tasks · one API call each · reasoning effort high

Can a model execute an ABS waterfall to the cent?

WaterfallBench tests whether a model can read a securitization's legal documents and compute its monthly payments to investors exactly as the trustee did, to the cent. The answer key is what the trustee actually paid, as filed with the SEC.

Loading results…

The task

What a waterfall is

An asset-backed security turns thousands of car loans or leases into bonds, sold in classes from senior (A) to junior (B, C). Every month the borrowers' payments are collected and paid out in a fixed order of priority, the waterfall:

Each step is paid only from the cash left after the steps above it. Simplified: each deal's agreements set the exact order, amounts and targets.

Why it matters

Securitization funds a large share of US car loans and leases: the 20 deals in this benchmark alone issued $23.5 billion of notes. The waterfall decides who gets paid and in what order, so an error moves cash between senior and junior investors, and because each month starts from the last month's balances, it carries forward. Today this is done with deal-specific cash-flow models built by hand from the documents. WaterfallBench asks whether a model can read the documents and get it right on its own.

What the model gets

A deal's first distribution date. Opening balances come from the prospectus, so there is no prior report to copy from. All documents are EDGAR HTML as filed:

InputFiled asWhat the waterfall needs from it
Final prospectus424(b)(5)Note classes, coupons, day counts, the priority of payments
Closing agreements8-K exhibitsIndenture and servicing agreements: the fee, reserve and overcollateralization definitions
Servicer report10-D Ex. 99.1That month's collections, pool balances, rates and dates, with every waterfall result removed

It must return the servicing fee, each class's interest, principal and ending balance, the reserve balance and the residual, in integer cents. Here is the report from a real task as the model sees it:

Servicer reportwaterfall results removed
Nissan Auto Lease 2026-B · Aug 2026
SummaryCouponInitial balanceBeginningEnding
Total Portfolio1,447,527,618.001,447,527,618.001,428,611,202.92
Class A-1 Notes3.949000%127,200,000.00127,200,000.00REMOVED
Class A-2a Notes4.440000%414,000,000.00414,000,000.00REMOVED
Class A-2b Notes3.978070%80,000,000.0080,000,000.00REMOVED
Class A-3 Notes4.690000%494,000,000.00494,000,000.00REMOVED
Class A-4 Notes4.780000%84,800,000.0084,800,000.00REMOVED
Class B Notes4.870000%49,940,000.0049,940,000.00REMOVED
Class C Notes5.080000%39,800,000.0039,800,000.00REMOVED
III. Distributions
Total Collections33,734,591.74
2. Reimbursement of Payment Advance0.00
4. Servicing FeeREMOVED
5. Class A-1 Notes Monthly InterestREMOVED
   Class A-1 Interest Carryover Shortfall0.00

Excerpt of fw-nissanlease-2026-b-2026-07. The same packet holds the 6.0 MB prospectus and 9 closing agreements.

Dataset

Download from Hugging Face

Tokens: mean prompt per task, Claude tokenizer, as sent by the harness. No task needed truncation.

Release checks

Every task passed these checks: the report's classes match the prospectus; each class's opening balance minus principal paid equals its ending balance, to the cent; class payments add up to the report's totals; no graded value is left in the packet; and the true answer grades 100% while an all-zeros answer resolves nothing.

Methodology

Harness

One API call per task: a fixed system prompt, the documents and the task. No tools, no code execution, no second turn, so the comparison is the model alone. EDGAR HTML is converted to text with table rows kept as cell | cell, then packed (report, prospectus, agreements) under a 1M-token cap; no task needed truncation. Reasoning effort is high for every model.

Evaluation metrics

Every task uses the same answer schema, given in the prompt. The model's reply is parsed, its classes are matched to the deal's by name, and every field is compared with the filed value, exactly.

In the promptanswer_schema.json
same for all 20 tasks
{
  "type": "object", "additionalProperties": false,
  "required": ["servicing_fee_paid", "reserve_ending_balance",
               "released_to_residual", "classes"],
  "properties": {
    "servicing_fee_paid":     {"type": "integer", "description": "US cents"},
    "reserve_ending_balance": {"type": "integer", "description": "US cents"},
    "released_to_residual":   {"type": "integer", "description": "US cents"},
    "classes": {"type": "array", "minItems": 1,
      "description": "one entry per class of notes the trust issued",
      "items": {"type": "object",
        "required": ["class", "interest_paid",
                     "principal_paid", "ending_balance"],
        "properties": {
          "class": {"type": "string",
            "description": "the class name as the prospectus gives it, e.g. A-1"},
          "interest_paid":  {"type": "integer", "description": "US cents"},
          "principal_paid": {"type": "integer", "description": "US cents"},
          "ending_balance": {"type": "integer", "description": "US cents"}}}}}
}
Actual output, gradedKimi K3.1 Pro
Nissan Auto Lease 2026-B
{
  "servicing_fee_paid": ✓,
  "reserve_ending_balance": ✓,
  "released_to_residual": ✗,
  "classes": [
    {"class": "A-1",  "interest_paid": ✓,
     "principal_paid": ✓, "ending_balance": ✓},
    {"class": "A-2a", "interest_paid": ✓,
     "principal_paid": ✓, "ending_balance": ✓},
    {"class": "A-2b", "interest_paid": ✓,
     "principal_paid": ✓, "ending_balance": ✓},
    {"class": "A-3",  "interest_paid": ✗,
     "principal_paid": ✓, "ending_balance": ✓},
    {"class": "A-4",  "interest_paid": ✓,
     "principal_paid": ✓, "ending_balance": ✓},
    {"class": "B",    "interest_paid": ✗,
     "principal_paid": ✓, "ending_balance": ✓},
    {"class": "C",    "interest_paid": ✓,
     "principal_paid": ✓, "ending_balance": ✓}
  ]
}

Values hidden: answer keys stay private. ✓ exact to the cent · ✗ wrong · ✓ trivial field (zero, or an opening balance left untouched).

MetricDefinitionThis output
Fully resolved primary1 if every graded field is exact to the cent, else 00
Field accuracyShare of non-trivial fields exactly right. Trivial fields are left out so untouched balances don't inflate the score9 / 12 = 75%
All fieldsShare of all graded fields exactly right (the task-by-task grid)21 / 24 = 88%

Leaderboard scores average these over the 20 tasks; ± is one standard error. Grading is deterministic, with no LLM judge. Cost uses each provider's list price for the tokens it reported.

Not graded. Auditing the runs turned up three places where the documents cannot determine a value. John Deere's first fee needs a pool balance no document prints. CarMax 2025-2's report departs from its own agreement's overcollateralization target. World Omni's agreement prorates the first fee without a day count. Those fields are left out of scoring; the filed values are never changed.

valkyrie run start --benchmark waterfallbench --dataset full-html \
  --agent waterfallbench-direct --model anthropic/claude-haiku-5-5 \
  -k effort high -s ANTHROPIC_API_KEY anthropic-api-key --concurrency 5

Results

Field accuracy vs task completion

Task by task

Green means every graded field is exact to the cent. Otherwise the cell shows the share of graded fields the model got exactly right.