Approach

What we measure, and where it stops.

Evidence about AI models is only useful if it survives scrutiny. This page sets out what our work can show, the standards we hold it to, and its limits.

01

What we measure

We test whether a specific AI model reproduces a specific copyrighted work when it is asked in ordinary ways, and we document every step so the result can be checked. Reproduction is what we measure. Whether a work was in a model's training data is a separate question, and a report speaks to it only as far as the evidence allows.

02

The standards we hold the work to

  • Fixed in advance. The test plan for an engagement is agreed before any result is seen, and it is not changed to reach a result.
  • Compared against a baseline. Results are read against works the model could not have seen, so a reader can tell a real signal from noise.
  • Error rates stated or marked unmeasured. Where an error rate has been measured for that model and type of work, the report states it and how. Where it has not, the report says so.
  • Repeatable. The record lets another expert run the same test and check the result.
  • Independent. Our conclusions are our own. Who retained and paid for the work is stated in every report.
03

Where the test is weak

  • Ordinary works. Models reproduce famous, widely copied works far more readily than ordinary ones. For many works a model was trained on, a test like ours may show nothing. We assess this with counsel before an engagement starts.
  • A negative result. When a model does not reproduce a work, that is not evidence the work was absent from training.
  • Closed models. For a model whose training data is not public, membership usually cannot be checked directly, and the report limits its conclusions accordingly.
  • Refusals and filters. Some models and hosts decline or filter output. A refusal is reported as a refusal.
  • Paraphrase. Close paraphrase is harder to establish than word-for-word reproduction.
  • Scope. Our measured work to date is on English-language text. Other languages and media are scoped case by case.
04

What this is not

  • Not legal advice, and not a guarantee of any outcome.
  • No claim of prior court acceptance. Whether any evidence is admitted is for the court to decide.
  • IPLYR staff do not act as testifying experts. We support the expert you retain, who reaches and states their own opinions.
05

Disclosure

Our approach builds on published research on extractable memorization. The full method is shared with your retained expert, and with opposing experts under protective order, so it can be examined where it matters. Common questions are answered in the FAQ.

Discuss a matter

We start with a scoping call with counsel: the works, the models, and whether a test can answer the question being asked.

Request a scoping call