Redwood v3 — Evidence — evaluated August 24, 2026

Evidence

How we measured it

Redwood v3 was evaluated on August 24, 2026 against the open model it is built on — same architecture, same 307M parameters, byte-identical weight-file size; Apache-2.0 lineage — on a separate 250-page synthetic evaluation corpus built for the redaction platform Redwood is being integrated into — a different generator from the model's own data, never used to build it, with the same settings for both. Every number on this page is measured; the ones that are worse for Redwood are in the tables, not the footnotes.

92%
fewer false detections on the evaluation corpus, both halves¹
100%
recall on the held-out half²
89%
fewer false detections on our own benchmark, full pipeline (model alone: 99.7%)³

The setup

The pages

250 single-page synthetic documents: case-record forms (60), incident narratives (80), agency letters (50), OCR-style two-column pages (40), FOIA-response pages built on public-domain federal boilerplate (20). Generated by the evaluation harness of the redaction platform Redwood is being integrated into — a different generator from the model's own data, never used to build it. 3,436 ground-truth PII spans across ten types, exact by construction.

case-record forms (60)
incident narratives (80)
agency letters (50)
OCR-style two-column pages (40)
FOIA-response pages built on public-domain federal boilerplate (20)

The pages were split once, by seeded random draw, into a tuning half and a held-out half. The held-out half never influenced the model or its settings; release gates are decided on it and the tuning half is reported alongside.

Tuning half (125)
Held-out half (125)

The conditions

Same settings for both models: in-process, on CPU, the identical full pipeline — model, deterministic patterns, repeat re-scan — over the identical pages, all ten types requested every time, run back to back on the same machine.

The comparison

The open model Redwood is built on — same architecture, same 307M parameters, byte-identical weight-file size; Apache-2.0 lineage — scored through the same pipeline on the same pages. Nothing but the model changes between the two columns, so the difference is the improvement. The base model is not named on this site.

What counts

A ground-truth span counts as found when detections of the same type cover at least half of its characters. A detection counts as a false detection when it does not cover at least half of any same-type ground-truth span. A detection wider than the truth is scored as correct, so boundary over-extension is not counted here. We report exact counts, not rates, wherever a count is small enough to list.

What a redaction looks like

The sample letter after Redwood, under a policy that redacts nine types and releases plain dates. The bars, the sources and the confidences are the pipeline's actual output on this letter; the receipt is an excerpt of the detection log it writes.

Response to records requestResponse to records request · Redacted copy

Re: Case No. [redacted: case number] FAM-2019-204418 · [detected, released by policy: date] March 4, 2026 detected · released by policy

Dear Ms. [redacted: person name] Dana Whitfield,

We have located the records you requested concerning [redacted: organization] Harbor Point Property Management. The subject, [redacted: person name] Marcus T. Ellery (DOB [redacted: date of birth] 06/12/1981, SSN [redacted: social security number] 000-45-6789), was contacted at [redacted: phone number] (503) 555-0142 and [redacted: email address] m.ellery@example.net. Copies were mailed to [redacted: full address] 1420 Larkspur Ave, Suite 3B, Portland, OR 97205.

The remaining pages are released in full under the Freedom of Information Act not PII · released.

cc: [redacted: person name] Marcus T. Ellery

Page 1 of 2

audit.json · detected 11 · redacted 10 · released_by_policy 1

modelperson nameDana Whitfield1.000
regexfull address1420 Larkspur Ave, Suite 3B, Portland, OR 972051.000
rescanperson nameMarcus T. Ellery · 2nd occurrence1.000
releaseddateMarch 4, 2026 · policy1.000

Person
Address
Organization
SSN
Phone
Email
Credit card
Date of birth
Date
Case number

Held-out half — the gate

False detections by PII type — Held-out half (1,702 spans) — base model vs Redwood v3, full pipeline, same settings, August 24, 2026

Base modelRedwood v3

Held-out half (1,702 spans) — base model vs Redwood v3, full pipeline, same settings, August 24, 2026

Scrolls sideways on narrow screens.

Held-out half (1,702 spans) — base model vs Redwood v3, full pipeline, same settings, August 24, 2026
PII type	Ground-truth spans	Base recall	Base false detections	Redwood recall	Redwood false detections	Δ false detections
Person	407	1.000	0	1.000	0	0
Address	218	1.000	0	1.000	0	0
Organization	275	1.000	45	1.000	14	−31
SSN	144	1.000	0	1.000	0	0
Phone	104	1.000	0	1.000	0	0
Email	103	1.000	0	1.000	0	0
Credit card	62	1.000	0	1.000	0	0
Date of birth	107	1.000	64	1.000	0	−64
Date	109	1.000	51	1.000	2	−49
Case number	173	1.000	0	1.000	0	0
Total	1702	1.000	160	1.000	16	−144

Recall and exact false-detection counts per type. Both models run through the identical pipeline on the identical pages.

Tuning half — reported, not gated

False detections by PII type — Tuning half (1,734 spans) — base model vs Redwood v3, full pipeline, same settings

Base modelRedwood v3

Tuning half (1,734 spans) — base model vs Redwood v3, full pipeline, same settings

Scrolls sideways on narrow screens.

Tuning half (1,734 spans) — base model vs Redwood v3, full pipeline, same settings
PII type	Base recall	Base false detections	Redwood recall	Redwood false detections	Δ false detections
Person. The one miss: a name in a signature block that every model tested, including the base model, also misses.	0.998	0	0.998	0	0
Address	1.000	0	1.000	0	0
Organization	1.000	17	1.000	5	−12
SSN	1.000	0	1.000	0	0
Phone	1.000	0	1.000	0	0
Email	1.000	0	1.000	0	0
Credit card	1.000	0	1.000	0	0
Date of birth	1.000	49	1.000	0	−49
Date	1.000	39	1.000	1	−38
Case number	1.000	0	1.000	0	0
Total	0.999	105	0.999	6	−99

This half informed error analysis during development, which is why the gate uses the other half. No evaluation page was used to build the model.

Every remaining false detection, named

All 22 false detections Redwood v3 produced across both halves, by class. None of them leaks personal information; each is counted as an error by the scoring rule all the same.

19
Organization
14 on the held-out half · 5 on the tuning half
What: agency sub-unit names in FOIA letterheads and routing headers.
- Records Management Division
- Office of Information Policy
- Freedom of Information Act Program
Why it counts: the ground truth marks only the top-line agency; these are redacted letterhead text, not leaked personal information. The bare statute name "Freedom of Information Act" is not flagged; the sub-unit "Freedom of Information Act Program" is.
3
Date
2 on the held-out half · 1 on the tuning half
What: "DOB mm/dd/yyyy" lines tagged as both date of birth and date.
Why it counts: the span is redacted either way; the second tag is counted as a false detection by the scoring rule.

Our own benchmark, with its caveats

We also keep our own benchmark: 483 passages — 437 across 14 document types plus 46 zero-PII memos — built to be hard. It is a vendor benchmark, so it comes second on this page.³

Our benchmark, model only — base model vs Redwood, overlap matching, same settings

Scrolls sideways on narrow screens.

Our benchmark, model only — base model vs Redwood, overlap matching, same settings
PII type	Base recall	Base false detections	Redwood recall	Redwood false detections
Person	0.780	6	0.828	0
Address	0.403	0	0.403	0
Organization	0.782	179	0.784	0
SSN	1.000	9	1.000	0
Phone	0.980	1	1.000	2
Email	0.884	0	0.884	0
Credit card	1.000	0	1.000	0
Date of birth	1.000	8	1.000	0
Date	0.999	419	0.999	0
Case number	0.573	42	0.609	0
All types, model only	0.797	664	0.812	2
All types, full pipeline	0.902	712	0.928	76

Micro-averaged across all spans. Model-only false detections fall 99.7%; with every pipeline layer on, 89% — the pipeline number is the one that describes what ships.

What the detection log contains

Every detection run writes one JSON file beside its output. Abridged from the sample letter — the real file also records the input path and every span, not just one per layer. This is the detection log; the export bundle's redaction log is a separate file that records positions and decisions and never the text.

{
  "model": "./models/redwood-v3",
  "threshold": 0.3,
  "policy_redact_labels": ["person name", "full address", "organization", "social security number", "phone number", "email address", "credit card number", "date of birth", "case number"],
  "counts": {
    "detected": 11,
    "redacted": 10,
    "released_by_policy": 1
  },
  "redacted": [
    {
      "label": "case number",
      "text": "FAM-2019-204418",
      "start": 42,
      "end": 57,
      "confidence": 1.0,
      "source": "model"
    },
    {
      "label": "full address",
      "text": "1420 Larkspur Ave, Suite 3B, Portland, OR 97205",
      "start": 333,
      "end": 380,
      "confidence": 1.0,
      "source": "regex"
    },
    {
      "label": "person name",
      "text": "Marcus T. Ellery",
      "start": 467,
      "end": 483,
      "confidence": 1.0,
      "source": "rescan"
    }
  ],
  "released_by_policy": [
    {
      "label": "date",
      "text": "March 4, 2026",
      "start": 60,
      "end": 73,
      "confidence": 1.0,
      "source": "model"
    }
  ]
}

source: Which layer produced the span: model, regex (deterministic pattern, always confidence 1.0) or rescan (a repeat of a span found earlier, carrying that span's confidence).
counts: detected spans, how many were redacted, and how many were detected but released_by_policy — the policy is an explicit list of labels to redact; there is no default.
text: The detection log contains the redacted text itself. Store it under the same controls as the records it describes.
confidence: Model confidences are near-saturated on this corpus (most spans score above 0.99), so the operating threshold is not a tuning dial.

Versions and the release gate

v1
First release. Cut false detections on our own benchmark by an order of magnitude; introduced 16 case-number false detections on the held-out half.
v2
Removed the case-number false detections; 6 false detections on the held-out half.
v3
Production. Organization recall rose from 0.971 to 1.000 with no other type dropping; 16 false detections on the held-out half, every one a letterhead sub-unit or a date of birth also tagged as a date.

No v4 is planned. Changes are driven by failures observed in production, not by a release calendar.

What Redwood does not do

The model is never shipped alone. The deterministic layer is load-bearing for case numbers and for long street addresses; every number on this page is the full pipeline.
Agency sub-unit lines in letterheads ("Records Management Division") may be redacted as organizations. This is the residual false-detection class above; it is redaction of letterhead text, not a leak.
Team and mascot names can be tagged as organizations on newsletter-like text.
Measured in English only, on text input. OCR quality is upstream of the model and is not measured here.
A ground-truth span counts as found at 50% character coverage; a detection wider than the truth counts as correct. Boundary precision is not scored.
One evaluation corpus, one annotation convention, both synthetic. Results on your records will differ — that is what a pilot measures.

Reproduce it

Results on other document collections will differ. A pilot runs Redwood on a sample of your own records, inside your environment, and returns the same scorecard format shown on this page.

Request the evaluation bundle Request a pilot

The evaluation bundle contains the per-type scorecards for each corpus half, plus the miss lists, behind every table on this page.

Notes

False detections — spans redacted that the ground truth does not mark; a detection wider than the truth counts as correct — measured with the full pipeline (model, deterministic patterns, repeat re-scan) at the same settings on the same pages. Held-out half: 160 with the open model Redwood is built on (Apache-2.0 lineage; same architecture, same 307M parameters, byte-identical weight-file size) → 16 with Redwood v3 (−90%); precision 0.914 → 0.991. Both halves combined: 265 → 22 (−92%). Of the 22, 19 are agency sub-unit names in letterheads and 3 are dates of birth also tagged as dates — redacted letterhead text and duplicate labels, not leaked text.↩
Measured on a separate 250-page synthetic evaluation corpus built for the redaction platform Redwood is being integrated into — a different generator from the model's own data, never used to build it, with 3,436 ground-truth PII spans across ten types, exact by construction. The pages were split once, by seeded random draw, into a 125-page tuning half and a 125-page held-out half that never influenced the model or its settings. On the held-out half both Redwood v3 and the open model it is built on found 1,702 of 1,702 spans (recall 100%). Across both halves Redwood found 3,435 of 3,436 (99.97%); the single miss is one name in a signature block that every model we tested, including the unmodified base model, also misses. A span counts as found when detections of the same type cover at least half of its characters.↩
Our own synthetic benchmark: 483 passages — 437 across 14 document types plus 46 zero-PII memos — overlap matching, same settings. Full pipeline: 712 → 76 false detections (−89%), recall 0.902 → 0.928. Model alone: 664 → 2 (−99.7%), recall 0.797 → 0.812. Scoring here differs from the corpus rule: each detection is matched one-to-one to a single ground-truth span, so a duplicate detection of an already-matched span counts as a false detection — stricter than the corpus rule, and why re-scan and address-pattern duplicates count here but not on the corpus. It is a vendor benchmark, deliberately hard; we lead with the numbers from the separate evaluation corpus.↩a ↩b

How we measured it

The setup

The pages

The conditions

The comparison

What counts

What a redaction looks like

Held-out half — the gate

Tuning half — reported, not gated

Every remaining false detection, named

Organization

Date

Our own benchmark, with its caveats

What the detection log contains

Versions and the release gate

What Redwood does not do

Reproduce it