Files
template-app/modules/knowledge
2026-11-14 09:56:45 +00:00
..
2026-11-14 09:56:45 +00:00
2026-11-14 09:56:45 +00:00
2026-11-14 09:56:45 +00:00
2026-11-14 09:56:45 +00:00
2026-11-14 09:56:45 +00:00
2026-11-14 09:56:45 +00:00
2026-11-14 09:56:45 +00:00
2026-11-14 09:56:45 +00:00
2026-11-14 09:56:45 +00:00
2026-11-14 09:56:45 +00:00

knowledge

The retrieval module: named collections of the app's documents, an idempotent document-AI enablement flow (knowledge.settings.enableDocumentAi), durable index/remove jobs, and cited search over the platform's knowledge service.

Events

A web request writes the document row and publishes knowledge.documentIndexRequested or knowledge.documentRemoveRequested in its transaction; the module's subscribers, run by the worker's outbox relay, turn each into the knowledge.index or knowledge.remove job. A deleted file (files.FileDeleted) cancels every document that indexed it and asks for its removal, carrying the storage key the file had: its files row is gone by then, so the knowledge.remove job removes the platform document by that key's Blob pathname rather than looking the file up (F407). A removal a person asks for reads the stored file's key when it is asked for, so a file deleted before the job runs is still removed by that key; a person's removal of a file already being deleted is refused (409), and the deletion's own removal does it (F407, review R1). The relay delivers in the organization's scope with a system actor, so a subscriber takes the organization from the delivery and only the workspace and project from the record, and refuses a record of another organization (adapters/jobs.ts). test/relay-scope.test.ts runs these deliveries through the worker's own relay on PostgreSQL.

The search backend

Retrieval is one platform service (ADR-061): an app keeps no vector store and no embedding pipeline of its own. The web shell and the worker bind one KnowledgeSearchService adapter, platform (KNOWLEDGE_BACKEND, the default and the only value; anything else stops the app at start, by the variable's name):

  • The adapter calls the platform's managed documents-index service through the kit-platform Knowledge client, with a deployment token that holds knowledge.collections:use (conusai.json asks for it, in every profile's copy that has a token block; * never carries it). The service's document_ref is the object's pathname inside the project's Blob prefix and options.content_sha256 is required. The adapter hashes the job's bytes, resolves that pathname through the files module's scoped storage reference, and maps returned citations back through the same project scope. Foreign or stale references are omitted; public file views still do not expose storage keys. The service keeps a collection per platform project and environment and answers 404 for one it does not hold, so the first index creates it in CONUSAI_ENVIRONMENT, and a search of a collection nothing was indexed into answers no results.
  • A refusal of the service is one closed set of codes, KNOWLEDGE_REFUSALS (adapters/refusals.ts): plan_required, feature_disabled, quota_exhausted, budget_exhausted, scope_missing (the token lacks knowledge.collections:use), indexing, approval_required, runtime_unavailable. knowledge.status carries it as service.refusal, knowledge.search answers it as the problem's code, and the pages show knowledge.refusals.<code> in English or Lithuanian. An index job records one that waiting cannot fix on the document's row instead of retrying it.

The module is off for an organization until an administrator enables document AI: with nothing bound to KNOWLEDGE_FLAGS, the knowledge.enabled flag answers the organization's knowledge_config row.

The local vectors backend (a linked Qdrant or pgvector, embeddings through the AI gateway) was removed: no profile needs it, since this module runs on postgres alone and ADR-061 gives retrieval to the platform. knowledge_config keeps the columns that backend used (source, service_id, url): the migration has run, and nothing reads them.

The live document list

The module declares the live collection knowledge.document (contract/collections.ts: table knowledge_document, key documentId, batch-get knowledge.document.getMany under knowledge.knowledge.search, at most 100 ids). Each collection's list on the page keeps the loader's rows as the seed of a liveCollection (ui/live-documents.tsx), filtered by the collection's name since a live topic is the whole project. A document another member asks to index appears, and each moves from queued to indexed or failed as the worker and the platform answer. Indexing is a useWrite that resolves once the live stream has seen the row; nothing re-runs the page's loader after a write (test/live-writes.test.ts).

Automatic PDF indexing (core-11)

An administrator can opt a project in on its knowledge page (knowledge.autoindex.set, permission knowledge.knowledge.manage), naming an existing collection. The opt-in is one row of knowledge_auto_index (migration 0002) that records who turned it on; no row is off.

The module contributes a files.processors processor for application/pdf, so the files module enqueues knowledge.autoindex once per stored PDF, in the upload's project (adapters/jobs.ts). The job reads nothing for a project that is not opted in, has knowledge off, or whose file is gone or no longer has the upload's digest. Otherwise it makes a knowledge_document row requested by the recorded administrator and runs the index lifecycle with that row (core/index-job.ts, indexPassages):

  • The job runs for nobody, so the worker's native admission checks the administrator's knowledge.knowledge.manage and the platform's OCR and knowledge services again before each step that leaves the process (apps/worker/src/native-admission.ts).
  • Before the read is paid for, the organization's eligibility and indexing quota are checked again through KNOWLEDGE_ENABLEMENT, as the enablement flow checks them. A refusal is recorded as plan_required, feature_disabled or quota_exhausted. There is no platform pre-check API for the quota yet, so today the knowledge service's own refusal at call time is the hard limit, as it is for a person's manual index.
  • The platform's document reader reads the PDF once per file and digest, and @conusai-sdk/ocr/passages cuts it into passages of at most the service's 64 KiB per item (adapters/reader.ts). A failed read, or a block id no metadata value can cite, is recorded as not_indexable.
  • Each passage is one inline item of the platform's knowledge service, under the file's Blob reference plus #p{page}.{index}, in the project's tenant partition, with metadata page, block_ids and file_id (adapters/platform-knowledge.ts, platformPassageIndex). A search hit on a passage cites its file, its page and its first block.
  • Before anything is sent, the file's reference and the count of passages per page are recorded in knowledge_passage_set (migration 0003; references and counts only, no text). Removing the document, as a deleted file does, tombstones every passage under that reference, even after the file itself is gone or the service refused the run partway.

The worker binds KNOWLEDGE_READER and KNOWLEDGE_PASSAGES lazily; the web shell binds neither, as a request never reads a PDF.

Why the module is Postgres-only (no Convex profile yet)

module.convex.ts ships a fail-closed Convex binding for a future wiring, but the module is not registered in the Convex profile (convex/lib/runtime.ts), and the manifest stays profiles: ["postgres"]. The integration review (2026-10-23) kept it Postgres-only because the module genuinely cannot run on the Convex runtime today:

  1. knowledge.search is an effect procedure — every backend leaves the process (the managed service over HTTP, or embeddings through the AI gateway plus a vector store). Convex queries/mutations cannot perform network I/O; effects need an explicit action composition like the files module's action→mutation workflows, and none exists for search.
  2. The index/remove jobs must run where the platform SDKs can leave the process. The Convex binding fails closed (… must run through the Convex Node action boundary); unlike documents' documentsActions:process, no composed Node action exists for these jobs yet, so every indexing request on the Convex profile would dispatch straight into the fence.
  3. knowledge.settings.enableDocumentAi is an effect: its use case performs network I/O (eligibility, quota), impossible inside a Convex mutation, so it needs its own action composition.
  4. knowledge.status takes its readiness snapshot from KNOWLEDGE_READINESS, which the Convex container does not bind, so it would answer unavailable.

Wiring the profile means composing those Node actions first (mirror convex/documentsActions.ts); registering the module before then would ship a broken search, a broken enablement flow and always-failing jobs, not a fail-closed gate.