ConusAI vendor v0.1.0-beta.114
@@ -0,0 +1,124 @@
|
||||
# knowledge
|
||||
|
||||
The retrieval module: named collections of the app's documents, an idempotent document-AI
|
||||
enablement flow (`knowledge.settings.enableDocumentAi`), durable index/remove jobs, and cited
|
||||
search over the platform's knowledge service.
|
||||
|
||||
## Events
|
||||
|
||||
A web request writes the document row and publishes `knowledge.documentIndexRequested` or
|
||||
`knowledge.documentRemoveRequested` in its transaction; the module's subscribers, run by the worker's
|
||||
outbox relay, turn each into the `knowledge.index` or `knowledge.remove` job. A deleted file
|
||||
(`files.FileDeleted`) cancels every document that indexed it and asks for its removal, carrying the storage key the
|
||||
file had: its `files` row is gone by then, so the `knowledge.remove` job removes the platform document by that key's
|
||||
Blob pathname rather than looking the file up (F407). A removal a person asks for reads the stored file's key when
|
||||
it is asked for, so a file deleted before the job runs is still removed by that key; a person's removal of a file
|
||||
already being deleted is refused (409), and the deletion's own removal does it (F407, review R1). The relay delivers
|
||||
in the organization's scope with a system actor, so a subscriber takes the organization from the delivery
|
||||
and only the workspace and project from the record, and refuses a record of another organization
|
||||
(`adapters/jobs.ts`). `test/relay-scope.test.ts` runs these deliveries through the worker's own relay on
|
||||
PostgreSQL.
|
||||
|
||||
## The search backend
|
||||
|
||||
Retrieval is one platform service (ADR-061): an app keeps no vector store and no embedding pipeline
|
||||
of its own. The web shell and the worker bind one `KnowledgeSearchService` adapter, `platform`
|
||||
(`KNOWLEDGE_BACKEND`, the default and the only value; anything else stops the app at start, by the
|
||||
variable's name):
|
||||
|
||||
- The adapter calls the platform's managed documents-index service through the kit-platform
|
||||
`Knowledge` client, with a deployment token that holds `knowledge.collections:use` (`conusai.json` asks for it,
|
||||
in every profile's copy that has a token block; `*` never carries it). The service's
|
||||
`document_ref` is the object's pathname inside the project's Blob prefix and
|
||||
`options.content_sha256` is required. The adapter hashes the job's bytes, resolves that pathname
|
||||
through the files module's scoped storage reference, and maps returned citations back through the
|
||||
same project scope. Foreign or stale references are omitted; public file views still do not expose
|
||||
storage keys. The service keeps a collection per platform project and environment and answers 404 for
|
||||
one it does not hold, so the first index creates it in `CONUSAI_ENVIRONMENT`, and a search of a
|
||||
collection nothing was indexed into answers no results.
|
||||
- A refusal of the service is one closed set of codes, `KNOWLEDGE_REFUSALS`
|
||||
(`adapters/refusals.ts`): `plan_required`, `feature_disabled`, `quota_exhausted`,
|
||||
`budget_exhausted`, `scope_missing` (the token lacks `knowledge.collections:use`), `indexing`,
|
||||
`approval_required`, `runtime_unavailable`. `knowledge.status` carries it as `service.refusal`,
|
||||
`knowledge.search` answers it as the problem's `code`, and the pages show `knowledge.refusals.<code>`
|
||||
in English or Lithuanian. An index job records one that waiting cannot fix on the document's row
|
||||
instead of retrying it.
|
||||
|
||||
The module is off for an organization until an administrator enables document AI: with nothing bound
|
||||
to `KNOWLEDGE_FLAGS`, the `knowledge.enabled` flag answers the organization's `knowledge_config` row.
|
||||
|
||||
The local `vectors` backend (a linked Qdrant or pgvector, embeddings through the AI gateway) was
|
||||
removed: no profile needs it, since this module runs on `postgres` alone and ADR-061 gives retrieval to
|
||||
the platform. `knowledge_config` keeps the columns that backend used (`source`, `service_id`, `url`):
|
||||
the migration has run, and nothing reads them.
|
||||
|
||||
## The live document list
|
||||
|
||||
The module declares the live collection `knowledge.document` (`contract/collections.ts`: table
|
||||
`knowledge_document`, key `documentId`, batch-get `knowledge.document.getMany` under
|
||||
`knowledge.knowledge.search`, at most 100 ids). Each collection's list on the page keeps the loader's
|
||||
rows as the seed of a `liveCollection` (`ui/live-documents.tsx`), filtered by the collection's name since
|
||||
a live topic is the whole project. A document another member asks to index appears, and each moves from
|
||||
queued to indexed or failed as the worker and the platform answer. Indexing is a `useWrite` that
|
||||
resolves once the live stream has seen the row; nothing re-runs the page's loader after a write
|
||||
(`test/live-writes.test.ts`).
|
||||
|
||||
## Automatic PDF indexing (core-11)
|
||||
|
||||
An administrator can opt a project in on its knowledge page (`knowledge.autoindex.set`, permission
|
||||
`knowledge.knowledge.manage`), naming an existing collection. The opt-in is one row of
|
||||
`knowledge_auto_index` (migration `0002`) that records who turned it on; no row is off.
|
||||
|
||||
The module contributes a `files.processors` processor for `application/pdf`, so the files module
|
||||
enqueues `knowledge.autoindex` once per stored PDF, in the upload's project (`adapters/jobs.ts`). The
|
||||
job reads nothing for a project that is not opted in, has knowledge off, or whose file is gone or no
|
||||
longer has the upload's digest. Otherwise it makes a `knowledge_document` row requested by the recorded
|
||||
administrator and runs the index lifecycle with that row (`core/index-job.ts`, `indexPassages`):
|
||||
|
||||
- The job runs for nobody, so the worker's native admission checks the administrator's
|
||||
`knowledge.knowledge.manage` and the platform's OCR and knowledge services again before each step
|
||||
that leaves the process (`apps/worker/src/native-admission.ts`).
|
||||
- Before the read is paid for, the organization's eligibility and indexing quota are checked again
|
||||
through `KNOWLEDGE_ENABLEMENT`, as the enablement flow checks them. A refusal is recorded as
|
||||
`plan_required`, `feature_disabled` or `quota_exhausted`. There is no platform pre-check API for the
|
||||
quota yet, so today the knowledge service's own refusal at call time is the hard limit, as it is for a
|
||||
person's manual index.
|
||||
- The platform's document reader reads the PDF once per file and digest, and `@conusai-sdk/ocr/passages`
|
||||
cuts it into passages of at most the service's 64 KiB per item (`adapters/reader.ts`). A failed read,
|
||||
or a block id no metadata value can cite, is recorded as `not_indexable`.
|
||||
- Each passage is one inline item of the platform's knowledge service, under the file's Blob reference
|
||||
plus `#p{page}.{index}`, in the project's tenant partition, with metadata `page`, `block_ids` and
|
||||
`file_id` (`adapters/platform-knowledge.ts`, `platformPassageIndex`). A search hit on a passage cites
|
||||
its file, its page and its first block.
|
||||
- Before anything is sent, the file's reference and the count of passages per page are recorded in
|
||||
`knowledge_passage_set` (migration `0003`; references and counts only, no text). Removing the
|
||||
document, as a deleted file does, tombstones every passage under that reference, even after the
|
||||
file itself is gone or the service refused the run partway.
|
||||
|
||||
The worker binds `KNOWLEDGE_READER` and `KNOWLEDGE_PASSAGES` lazily; the web shell binds neither, as a
|
||||
request never reads a PDF.
|
||||
|
||||
## Why the module is Postgres-only (no Convex profile yet)
|
||||
|
||||
`module.convex.ts` ships a fail-closed Convex binding for a future wiring, but the module is
|
||||
**not** registered in the Convex profile (`convex/lib/runtime.ts`), and the manifest stays
|
||||
`profiles: ["postgres"]`. The integration review (2026-10-23) kept it Postgres-only because the
|
||||
module genuinely cannot run on the Convex runtime today:
|
||||
|
||||
1. `knowledge.search` is an **effect** procedure — every backend leaves the process (the managed
|
||||
service over HTTP, or embeddings through the AI gateway plus a vector store). Convex
|
||||
queries/mutations cannot perform network I/O; effects need an explicit action composition like
|
||||
the files module's action→mutation workflows, and none exists for search.
|
||||
2. The index/remove **jobs** must run where the platform SDKs can leave the process. The Convex
|
||||
binding fails closed (`… must run through the Convex Node action boundary`); unlike documents'
|
||||
`documentsActions:process`, no composed Node action exists for these jobs yet, so every
|
||||
indexing request on the Convex profile would dispatch straight into the fence.
|
||||
3. `knowledge.settings.enableDocumentAi` is an `effect`: its use case performs network I/O
|
||||
(eligibility, quota), impossible inside a Convex mutation, so it needs
|
||||
its own action composition.
|
||||
4. `knowledge.status` takes its readiness snapshot from `KNOWLEDGE_READINESS`, which the Convex
|
||||
container does not bind, so it would answer `unavailable`.
|
||||
|
||||
Wiring the profile means composing those Node actions first (mirror `convex/documentsActions.ts`);
|
||||
registering the module before then would ship a broken search, a broken enablement flow and
|
||||
always-failing jobs, not a fail-closed gate.
|
||||
Reference in New Issue
Block a user