ConusAI vendor v0.1.0-beta.114

2026-11-14 09:56:45 +00:00
commit 1ce8d23c35
867 changed files with 122104 additions and 0 deletions
@@ -0,0 +1,89 @@
+<!-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0 -->
+# Evals
+
+An eval answers one question a unit test cannot: *did this change to a prompt, a schema or a model make
+the extraction better or worse?* A module that asks a model for something that changes records keeps a
+small golden set here, a scorer, and thresholds that live in the repository.
+
+## documents: line extraction
+
+`modules/documents` reads a document with OCR and asks a model, through the kit's `ModelGateway`, for the
+labelled lines of each page. `evals/documents` scores that extraction on eight invented documents
+(`golden/*.json`: an invoice, a specification sheet, a scanned delivery note and a page with no fields,
+in English and in Lithuanian; one of them carries a line that tries to give the reader an instruction;
+two are scanned PDFs, one with a drawing sheet the extraction must leave out and a running line the reader
+flags as repeated, one in Lithuanian with a blank page). A case may name the pages that must be skipped
+(`skipped`), the class the reader gave a page (`class`) and its repeated rows (`repeated`); offline, the
+pages the extractor shows the model as images are drawn through the fake platform's page route
+(`GET /v1/ocr/jobs/{id}/pages/{n}.png`), live there is no job to draw them from and a scan is read from its text.
+There is no real, customer or personal data in them, and a new case must keep it that way.
+
+It reports three numbers over the whole set:
+
+| Metric | Meaning |
+|---|---|
+| recall | of the lines a person expects, the share the extractor proposed (same page; same label and value once whitespace, case and diacritics are folded) |
+| precision | of the lines it proposed, the share that were expected |
+| verification | of the lines it proposed, the share whose quote is on the page it cites. It is computed again here from the case's text, not read from the extractor's own `verification` field |
+
+### Offline: the gate
+
+```bash
+pnpm eval          # the same as part of `pnpm test`
+```
+
+The golden set goes through the real extractor and the real gateway client to the fake platform
+(`@conusai-kit/testing`), whose model is a deterministic reader (`test/support/fake-model.ts`). This
+measures the module's own code -- the request it builds, the schema it parses, the verification, the
+scoring -- and not a model, so its thresholds are 100% (`thresholds.json`, `offline`). Three negative
+controls prove the eval can fail: quotes that are not on the page, a model that finds nothing, and one
+that invents lines each fail it. Add them for any new metric.
+
+### Both extractors
+
+The same golden set, scorer and offline thresholds also run on the platform's Document Items run
+(`platformItemsExtractor`, ADR-069), the extractor the `documents.platform_items` flag selects:
+`platform-items.eval.test.ts`. The fake platform plays the items run with the same deterministic reader,
+each item cited by its line and box, so the eval measures the module's side of the seam: the request, the
+mapping of `conusai.documents.items/v1` to lines, the second check of every quote against the page, and the
+skipped pages. Its negative controls (quotes off the page though the platform calls them exact, nothing
+found, an invented item) fail it. There is no live items eval here: the golden set is text, not files, so
+there is no job to run a profile over; a product runs its own against its private set.
+
+### Live: a real model
+
+```bash
+# After secure project linking has injected the instance URL and scoped token:
+AI_MODEL=anthropic/claude-sonnet-5 pnpm eval:live
+```
+
+Run this only after the new platform revision is installed and its source/bundle identity is verified.
+Use that instance's normal linked-project environment: `CONUSAI_API_URL` ends in `/api`, its scoped
+`CONUSAI_API_TOKEN` is injected securely without printing it or placing it in shell history, and
+`AI_MODEL` selects an enabled real model. A missing endpoint, token or model is a readiness blocker;
+the fake platform or an older instance cannot establish acceptance of the new deployment.
+
+The same cases and scorer against the platform's AI gateway, with the `live` thresholds of
+`thresholds.json` (recall 85%, precision 80%, verification 95%). It costs credits: run it when a prompt,
+the schema or the model changes, and put the numbers in the change. The report names each missed and
+each extra line.
+
+### promptfoo
+
+```bash
+CONUSAI_API_URL=... CONUSAI_API_TOKEN=... pnpm eval:promptfoo
+```
+
+`promptfooconfig.yaml` runs the same cases through [promptfoo](https://www.promptfoo.dev) (pinned in the
+script, started with `npx`, so it adds nothing to this app's dependencies), whose provider is the
+module's own extractor and whose assertion is the scorer above. Use it for its comparison view: list the
+provider again with another `config.model` to put two models side by side. `promptfoo view` shows the last run.
+
+### Changing the golden set or the prompt
+
+- A new case is a file in `golden/`, a `case` row in `promptfooconfig.yaml` (a test fails if the two
+  differ) and, if the deterministic reader cannot read it, a change to the reader in
+  `test/support/fake-model.ts` that makes the same promise the prompt does.
+- A change to `EXTRACT_SYSTEM_PROMPT` in `modules/documents/adapters/model-extractor.ts` bumps
+  `EXTRACT_PROMPT_VERSION`: the version is recorded on every proposal and is part of the cache key of a
+  model's reply, so an old reply is never served for a new prompt.