125 lines
8.7 KiB
Markdown
125 lines
8.7 KiB
Markdown
# knowledge
|
|
|
|
The retrieval module: named collections of the app's documents, an idempotent document-AI
|
|
enablement flow (`knowledge.settings.enableDocumentAi`), durable index/remove jobs, and cited
|
|
search over the platform's knowledge service.
|
|
|
|
## Events
|
|
|
|
A web request writes the document row and publishes `knowledge.documentIndexRequested` or
|
|
`knowledge.documentRemoveRequested` in its transaction; the module's subscribers, run by the worker's
|
|
outbox relay, turn each into the `knowledge.index` or `knowledge.remove` job. A deleted file
|
|
(`files.FileDeleted`) cancels every document that indexed it and asks for its removal, carrying the storage key the
|
|
file had: its `files` row is gone by then, so the `knowledge.remove` job removes the platform document by that key's
|
|
Blob pathname rather than looking the file up (F407). A removal a person asks for reads the stored file's key when
|
|
it is asked for, so a file deleted before the job runs is still removed by that key; a person's removal of a file
|
|
already being deleted is refused (409), and the deletion's own removal does it (F407, review R1). The relay delivers
|
|
in the organization's scope with a system actor, so a subscriber takes the organization from the delivery
|
|
and only the workspace and project from the record, and refuses a record of another organization
|
|
(`adapters/jobs.ts`). `test/relay-scope.test.ts` runs these deliveries through the worker's own relay on
|
|
PostgreSQL.
|
|
|
|
## The search backend
|
|
|
|
Retrieval is one platform service (ADR-061): an app keeps no vector store and no embedding pipeline
|
|
of its own. The web shell and the worker bind one `KnowledgeSearchService` adapter, `platform`
|
|
(`KNOWLEDGE_BACKEND`, the default and the only value; anything else stops the app at start, by the
|
|
variable's name):
|
|
|
|
- The adapter calls the platform's managed documents-index service through the kit-platform
|
|
`Knowledge` client, with a deployment token that holds `knowledge.collections:use` (`conusai.json` asks for it,
|
|
in every profile's copy that has a token block; `*` never carries it). The service's
|
|
`document_ref` is the object's pathname inside the project's Blob prefix and
|
|
`options.content_sha256` is required. The adapter hashes the job's bytes, resolves that pathname
|
|
through the files module's scoped storage reference, and maps returned citations back through the
|
|
same project scope. Foreign or stale references are omitted; public file views still do not expose
|
|
storage keys. The service keeps a collection per platform project and environment and answers 404 for
|
|
one it does not hold, so the first index creates it in `CONUSAI_ENVIRONMENT`, and a search of a
|
|
collection nothing was indexed into answers no results.
|
|
- A refusal of the service is one closed set of codes, `KNOWLEDGE_REFUSALS`
|
|
(`adapters/refusals.ts`): `plan_required`, `feature_disabled`, `quota_exhausted`,
|
|
`budget_exhausted`, `scope_missing` (the token lacks `knowledge.collections:use`), `indexing`,
|
|
`approval_required`, `runtime_unavailable`. `knowledge.status` carries it as `service.refusal`,
|
|
`knowledge.search` answers it as the problem's `code`, and the pages show `knowledge.refusals.<code>`
|
|
in English or Lithuanian. An index job records one that waiting cannot fix on the document's row
|
|
instead of retrying it.
|
|
|
|
The module is off for an organization until an administrator enables document AI: with nothing bound
|
|
to `KNOWLEDGE_FLAGS`, the `knowledge.enabled` flag answers the organization's `knowledge_config` row.
|
|
|
|
The local `vectors` backend (a linked Qdrant or pgvector, embeddings through the AI gateway) was
|
|
removed: no profile needs it, since this module runs on `postgres` alone and ADR-061 gives retrieval to
|
|
the platform. `knowledge_config` keeps the columns that backend used (`source`, `service_id`, `url`):
|
|
the migration has run, and nothing reads them.
|
|
|
|
## The live document list
|
|
|
|
The module declares the live collection `knowledge.document` (`contract/collections.ts`: table
|
|
`knowledge_document`, key `documentId`, batch-get `knowledge.document.getMany` under
|
|
`knowledge.knowledge.search`, at most 100 ids). Each collection's list on the page keeps the loader's
|
|
rows as the seed of a `liveCollection` (`ui/live-documents.tsx`), filtered by the collection's name since
|
|
a live topic is the whole project. A document another member asks to index appears, and each moves from
|
|
queued to indexed or failed as the worker and the platform answer. Indexing is a `useWrite` that
|
|
resolves once the live stream has seen the row; nothing re-runs the page's loader after a write
|
|
(`test/live-writes.test.ts`).
|
|
|
|
## Automatic PDF indexing (core-11)
|
|
|
|
An administrator can opt a project in on its knowledge page (`knowledge.autoindex.set`, permission
|
|
`knowledge.knowledge.manage`), naming an existing collection. The opt-in is one row of
|
|
`knowledge_auto_index` (migration `0002`) that records who turned it on; no row is off.
|
|
|
|
The module contributes a `files.processors` processor for `application/pdf`, so the files module
|
|
enqueues `knowledge.autoindex` once per stored PDF, in the upload's project (`adapters/jobs.ts`). The
|
|
job reads nothing for a project that is not opted in, has knowledge off, or whose file is gone or no
|
|
longer has the upload's digest. Otherwise it makes a `knowledge_document` row requested by the recorded
|
|
administrator and runs the index lifecycle with that row (`core/index-job.ts`, `indexPassages`):
|
|
|
|
- The job runs for nobody, so the worker's native admission checks the administrator's
|
|
`knowledge.knowledge.manage` and the platform's OCR and knowledge services again before each step
|
|
that leaves the process (`apps/worker/src/native-admission.ts`).
|
|
- Before the read is paid for, the organization's eligibility and indexing quota are checked again
|
|
through `KNOWLEDGE_ENABLEMENT`, as the enablement flow checks them. A refusal is recorded as
|
|
`plan_required`, `feature_disabled` or `quota_exhausted`. There is no platform pre-check API for the
|
|
quota yet, so today the knowledge service's own refusal at call time is the hard limit, as it is for a
|
|
person's manual index.
|
|
- The platform's document reader reads the PDF once per file and digest, and `@conusai-sdk/ocr/passages`
|
|
cuts it into passages of at most the service's 64 KiB per item (`adapters/reader.ts`). A failed read,
|
|
or a block id no metadata value can cite, is recorded as `not_indexable`.
|
|
- Each passage is one inline item of the platform's knowledge service, under the file's Blob reference
|
|
plus `#p{page}.{index}`, in the project's tenant partition, with metadata `page`, `block_ids` and
|
|
`file_id` (`adapters/platform-knowledge.ts`, `platformPassageIndex`). A search hit on a passage cites
|
|
its file, its page and its first block.
|
|
- Before anything is sent, the file's reference and the count of passages per page are recorded in
|
|
`knowledge_passage_set` (migration `0003`; references and counts only, no text). Removing the
|
|
document, as a deleted file does, tombstones every passage under that reference, even after the
|
|
file itself is gone or the service refused the run partway.
|
|
|
|
The worker binds `KNOWLEDGE_READER` and `KNOWLEDGE_PASSAGES` lazily; the web shell binds neither, as a
|
|
request never reads a PDF.
|
|
|
|
## Why the module is Postgres-only (no Convex profile yet)
|
|
|
|
`module.convex.ts` ships a fail-closed Convex binding for a future wiring, but the module is
|
|
**not** registered in the Convex profile (`convex/lib/runtime.ts`), and the manifest stays
|
|
`profiles: ["postgres"]`. The integration review (2026-10-23) kept it Postgres-only because the
|
|
module genuinely cannot run on the Convex runtime today:
|
|
|
|
1. `knowledge.search` is an **effect** procedure — every backend leaves the process (the managed
|
|
service over HTTP, or embeddings through the AI gateway plus a vector store). Convex
|
|
queries/mutations cannot perform network I/O; effects need an explicit action composition like
|
|
the files module's action→mutation workflows, and none exists for search.
|
|
2. The index/remove **jobs** must run where the platform SDKs can leave the process. The Convex
|
|
binding fails closed (`… must run through the Convex Node action boundary`); unlike documents'
|
|
`documentsActions:process`, no composed Node action exists for these jobs yet, so every
|
|
indexing request on the Convex profile would dispatch straight into the fence.
|
|
3. `knowledge.settings.enableDocumentAi` is an `effect`: its use case performs network I/O
|
|
(eligibility, quota), impossible inside a Convex mutation, so it needs
|
|
its own action composition.
|
|
4. `knowledge.status` takes its readiness snapshot from `KNOWLEDGE_READINESS`, which the Convex
|
|
container does not bind, so it would answer `unavailable`.
|
|
|
|
Wiring the profile means composing those Node actions first (mirror `convex/documentsActions.ts`);
|
|
registering the module before then would ship a broken search, a broken enablement flow and
|
|
always-failing jobs, not a fail-closed gate.
|