ConusAI vendor v0.1.0-beta.114
ci / gate (push) Waiting to run

This commit is contained in:
ConusAI vendor v0.1.0-beta.114
2026-11-14 09:56:45 +00:00
commit 1ce8d23c35
867 changed files with 122104 additions and 0 deletions
+124
View File
@@ -0,0 +1,124 @@
# knowledge
The retrieval module: named collections of the app's documents, an idempotent document-AI
enablement flow (`knowledge.settings.enableDocumentAi`), durable index/remove jobs, and cited
search over the platform's knowledge service.
## Events
A web request writes the document row and publishes `knowledge.documentIndexRequested` or
`knowledge.documentRemoveRequested` in its transaction; the module's subscribers, run by the worker's
outbox relay, turn each into the `knowledge.index` or `knowledge.remove` job. A deleted file
(`files.FileDeleted`) cancels every document that indexed it and asks for its removal, carrying the storage key the
file had: its `files` row is gone by then, so the `knowledge.remove` job removes the platform document by that key's
Blob pathname rather than looking the file up (F407). A removal a person asks for reads the stored file's key when
it is asked for, so a file deleted before the job runs is still removed by that key; a person's removal of a file
already being deleted is refused (409), and the deletion's own removal does it (F407, review R1). The relay delivers
in the organization's scope with a system actor, so a subscriber takes the organization from the delivery
and only the workspace and project from the record, and refuses a record of another organization
(`adapters/jobs.ts`). `test/relay-scope.test.ts` runs these deliveries through the worker's own relay on
PostgreSQL.
## The search backend
Retrieval is one platform service (ADR-061): an app keeps no vector store and no embedding pipeline
of its own. The web shell and the worker bind one `KnowledgeSearchService` adapter, `platform`
(`KNOWLEDGE_BACKEND`, the default and the only value; anything else stops the app at start, by the
variable's name):
- The adapter calls the platform's managed documents-index service through the kit-platform
`Knowledge` client, with a deployment token that holds `knowledge.collections:use` (`conusai.json` asks for it,
in every profile's copy that has a token block; `*` never carries it). The service's
`document_ref` is the object's pathname inside the project's Blob prefix and
`options.content_sha256` is required. The adapter hashes the job's bytes, resolves that pathname
through the files module's scoped storage reference, and maps returned citations back through the
same project scope. Foreign or stale references are omitted; public file views still do not expose
storage keys. The service keeps a collection per platform project and environment and answers 404 for
one it does not hold, so the first index creates it in `CONUSAI_ENVIRONMENT`, and a search of a
collection nothing was indexed into answers no results.
- A refusal of the service is one closed set of codes, `KNOWLEDGE_REFUSALS`
(`adapters/refusals.ts`): `plan_required`, `feature_disabled`, `quota_exhausted`,
`budget_exhausted`, `scope_missing` (the token lacks `knowledge.collections:use`), `indexing`,
`approval_required`, `runtime_unavailable`. `knowledge.status` carries it as `service.refusal`,
`knowledge.search` answers it as the problem's `code`, and the pages show `knowledge.refusals.<code>`
in English or Lithuanian. An index job records one that waiting cannot fix on the document's row
instead of retrying it.
The module is off for an organization until an administrator enables document AI: with nothing bound
to `KNOWLEDGE_FLAGS`, the `knowledge.enabled` flag answers the organization's `knowledge_config` row.
The local `vectors` backend (a linked Qdrant or pgvector, embeddings through the AI gateway) was
removed: no profile needs it, since this module runs on `postgres` alone and ADR-061 gives retrieval to
the platform. `knowledge_config` keeps the columns that backend used (`source`, `service_id`, `url`):
the migration has run, and nothing reads them.
## The live document list
The module declares the live collection `knowledge.document` (`contract/collections.ts`: table
`knowledge_document`, key `documentId`, batch-get `knowledge.document.getMany` under
`knowledge.knowledge.search`, at most 100 ids). Each collection's list on the page keeps the loader's
rows as the seed of a `liveCollection` (`ui/live-documents.tsx`), filtered by the collection's name since
a live topic is the whole project. A document another member asks to index appears, and each moves from
queued to indexed or failed as the worker and the platform answer. Indexing is a `useWrite` that
resolves once the live stream has seen the row; nothing re-runs the page's loader after a write
(`test/live-writes.test.ts`).
## Automatic PDF indexing (core-11)
An administrator can opt a project in on its knowledge page (`knowledge.autoindex.set`, permission
`knowledge.knowledge.manage`), naming an existing collection. The opt-in is one row of
`knowledge_auto_index` (migration `0002`) that records who turned it on; no row is off.
The module contributes a `files.processors` processor for `application/pdf`, so the files module
enqueues `knowledge.autoindex` once per stored PDF, in the upload's project (`adapters/jobs.ts`). The
job reads nothing for a project that is not opted in, has knowledge off, or whose file is gone or no
longer has the upload's digest. Otherwise it makes a `knowledge_document` row requested by the recorded
administrator and runs the index lifecycle with that row (`core/index-job.ts`, `indexPassages`):
- The job runs for nobody, so the worker's native admission checks the administrator's
`knowledge.knowledge.manage` and the platform's OCR and knowledge services again before each step
that leaves the process (`apps/worker/src/native-admission.ts`).
- Before the read is paid for, the organization's eligibility and indexing quota are checked again
through `KNOWLEDGE_ENABLEMENT`, as the enablement flow checks them. A refusal is recorded as
`plan_required`, `feature_disabled` or `quota_exhausted`. There is no platform pre-check API for the
quota yet, so today the knowledge service's own refusal at call time is the hard limit, as it is for a
person's manual index.
- The platform's document reader reads the PDF once per file and digest, and `@conusai-sdk/ocr/passages`
cuts it into passages of at most the service's 64 KiB per item (`adapters/reader.ts`). A failed read,
or a block id no metadata value can cite, is recorded as `not_indexable`.
- Each passage is one inline item of the platform's knowledge service, under the file's Blob reference
plus `#p{page}.{index}`, in the project's tenant partition, with metadata `page`, `block_ids` and
`file_id` (`adapters/platform-knowledge.ts`, `platformPassageIndex`). A search hit on a passage cites
its file, its page and its first block.
- Before anything is sent, the file's reference and the count of passages per page are recorded in
`knowledge_passage_set` (migration `0003`; references and counts only, no text). Removing the
document, as a deleted file does, tombstones every passage under that reference, even after the
file itself is gone or the service refused the run partway.
The worker binds `KNOWLEDGE_READER` and `KNOWLEDGE_PASSAGES` lazily; the web shell binds neither, as a
request never reads a PDF.
## Why the module is Postgres-only (no Convex profile yet)
`module.convex.ts` ships a fail-closed Convex binding for a future wiring, but the module is
**not** registered in the Convex profile (`convex/lib/runtime.ts`), and the manifest stays
`profiles: ["postgres"]`. The integration review (2026-10-23) kept it Postgres-only because the
module genuinely cannot run on the Convex runtime today:
1. `knowledge.search` is an **effect** procedure — every backend leaves the process (the managed
service over HTTP, or embeddings through the AI gateway plus a vector store). Convex
queries/mutations cannot perform network I/O; effects need an explicit action composition like
the files module's action→mutation workflows, and none exists for search.
2. The index/remove **jobs** must run where the platform SDKs can leave the process. The Convex
binding fails closed (`… must run through the Convex Node action boundary`); unlike documents'
`documentsActions:process`, no composed Node action exists for these jobs yet, so every
indexing request on the Convex profile would dispatch straight into the fence.
3. `knowledge.settings.enableDocumentAi` is an `effect`: its use case performs network I/O
(eligibility, quota), impossible inside a Convex mutation, so it needs
its own action composition.
4. `knowledge.status` takes its readiness snapshot from `KNOWLEDGE_READINESS`, which the Convex
container does not bind, so it would answer `unavailable`.
Wiring the profile means composing those Node actions first (mirror `convex/documentsActions.ts`);
registering the module before then would ship a broken search, a broken enablement flow and
always-failing jobs, not a fail-closed gate.
@@ -0,0 +1,62 @@
import { Container, type TenantScope } from "@conusai-kit/core"
import { createMemoryDataSource, DATA_SOURCE, SYSTEM_DATA } from "@conusai-kit/data"
import { accountData, projects } from "@conusai-kit/tenancy"
import { expect, it } from "vitest"
import { custodyModule } from "../../custody.js"
import { knowledgeAccountData } from "../adapters/account.js"
import { autoIndex, KNOWLEDGE_ENTITIES } from "../contract/index.js"
import { knowledgeModule } from "../module.js"
const subject = "01900000-0000-7000-0000-000000000009"
async function fixture() {
const data = createMemoryDataSource({ entities: [projects, ...KNOWLEDGE_ENTITIES] })
const container = new Container()
container.value(DATA_SOURCE, data)
container.value(SYSTEM_DATA, data)
const scopes: TenantScope[] = []
for (const n of [1, 2, 3]) {
const orgId = `01900000-0000-7000-0000-00000000000${n}`
const workspaceId = `01900000-0000-7000-0000-00000000001${n}`
const actor = { userId: subject }
const project = await data
.forScope({ level: "workspace", orgId, workspaceId, actor })
.of(projects)
.insert({ slug: "site", name: "Site", moduleKey: "notes", archivedAt: null })
const scope: TenantScope = { level: "project", orgId, workspaceId, projectId: project.id, actor }
scopes.push(scope)
await data
.forScope(scope)
.of(autoIndex)
.insert({
collectionName: "contracts",
enabledBy: n === 3 ? "another-person" : subject,
enabledAt: "2026-11-11T00:00:00.000Z",
})
}
return { data, container, scopes }
}
it("exports and erases only that administrator’s opt-ins across projects, idempotently", async () => {
const { data, container, scopes } = await fixture()
const exported = await knowledgeAccountData.export?.({ subject, container })
expect(exported?.automaticIndexing).toHaveLength(2)
await knowledgeAccountData.erase?.({ subject, container })
expect(await knowledgeAccountData.export?.({ subject, container })).toEqual({ automaticIndexing: [] })
await knowledgeAccountData.erase?.({ subject, container })
const otherScope = scopes[2]
if (!otherScope) throw new Error("Missing unrelated project fixture")
expect((await data.forScope(otherScope).of(autoIndex).list()).items).toHaveLength(1)
})
it("retains the account contribution when processing is deselected", () => {
const retained = custodyModule(knowledgeModule)
expect(retained.extends?.find((entry) => entry.point.id === accountData.id)?.values).toEqual([knowledgeAccountData])
})
it("an aborted account operation changes no opt-in", async () => {
const { data, container, scopes } = await fixture()
const controller = new AbortController()
controller.abort()
await expect(knowledgeAccountData.erase?.({ subject, container, signal: controller.signal })).rejects.toThrow()
for (const scope of scopes) expect((await data.forScope(scope).of(autoIndex).list()).items).toHaveLength(1)
})
@@ -0,0 +1,309 @@
// The automatic PDF intake (template lesson core-11): an uploaded PDF of an opted-in project is read by
// the platform's document reader and indexed passage by passage into the opted-in collection, for the
// administrator who turned it on, whose permission is checked again before each step that leaves the
// process. A project that is not opted in, or has knowledge off, reads nothing; removing the document
// tombstones every passage it indexed, even once the file itself is gone.
import type { TenantScope } from "@conusai-kit/core"
import { createMemoryDataSource } from "@conusai-kit/data"
import { DEFAULT_FILES_POLICY, files, filesProcessors, sha256Hex } from "@conusai-kit/files"
import { fakeObjectStore } from "@conusai-kit/testing"
import { describe, expect, it } from "vitest"
import {
AUTO_INDEX_JOB,
AUTO_INDEX_PROCESSOR_KEY,
autoIndex,
collections,
documents,
KNOWLEDGE_ENTITIES,
passageSets,
} from "../contract/index.js"
import { autoIndexUpload } from "../core/auto-index-job.js"
import { removeDocument } from "../core/index-job.js"
import type {
DocumentAiPlatform,
KnowledgeDeps,
KnowledgePassage,
KnowledgePassageIndex,
KnowledgeReader,
} from "../core/ports.js"
import { knowledgeModule } from "../module.js"
const ADMIN = "01900000-0000-7000-0000-000000000009"
const UPLOADER = "01900000-0000-7000-0000-000000000010"
const scope: TenantScope = {
level: "project",
actor: { userId: "system:knowledge.autoindex", system: true },
orgId: "01900000-0000-7000-0000-000000000001",
workspaceId: "01900000-0000-7000-0000-000000000002",
projectId: "01900000-0000-7000-0000-000000000003",
}
const pdf = new TextEncoder().encode("%PDF-1.7 synthetic")
const read: readonly KnowledgePassage[] = [
{ page: 1, index: 0, blockIds: ["b1", "b2"], text: "First page, first passage" },
{ page: 1, index: 1, blockIds: ["b3"], text: "First page, second passage" },
{ page: 3, index: 0, blockIds: [], text: "Third page" },
]
function makeDeps(
options: { enabled?: boolean; reader?: KnowledgeReader; indexError?: Error; enablement?: DocumentAiPlatform } = {},
) {
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES, files] })
const store = fakeObjectStore()
const calls = {
reads: [] as Array<{ filename: string; idempotencyKey: string }>,
indexed: [] as Array<{ collectionName: string; fileId: string; passages: readonly KnowledgePassage[] }>,
removed: [] as Array<{ documentRef: string; passages: ReadonlyArray<{ page: number; index: number }> }>,
admitted: [] as string[],
}
const reader: KnowledgeReader = options.reader ?? {
async read(bytes, input) {
expect(bytes).toEqual(pdf)
calls.reads.push({ filename: input.filename, idempotencyKey: input.idempotencyKey })
return read
},
}
const passages: KnowledgePassageIndex = {
async reference(_scope, fileId) {
return `org/${scope.orgId}/${fileRow.storageKey}?${fileId}`
},
async index(_scope, input) {
expect(input.documentRef).toBe(`org/${scope.orgId}/${fileRow.storageKey}?${input.fileId}`)
calls.indexed.push({ collectionName: input.collectionName, fileId: input.fileId, passages: input.passages })
if (options.indexError) throw options.indexError
},
async remove(_scope, input) {
calls.removed.push({ documentRef: input.documentRef, passages: input.passages })
},
}
const deps: KnowledgeDeps = {
data,
files: { data, store, policy: DEFAULT_FILES_POLICY },
flags: { enabled: async () => options.enabled ?? true },
reader,
passages,
enablement: options.enablement ?? {
checkEligibility: async () => ({ ok: true }),
indexingQuota: async () => 1000,
},
beforeIndex: async (subject) => {
calls.admitted.push(subject)
},
}
return { data, store, deps, calls }
}
const fileRow = {
status: "stored" as const,
storageKey: `files/${scope.workspaceId}/${scope.projectId}/01900000-0000-7000-0000-000000000100`,
filename: "contract.pdf",
contentType: "application/pdf",
size: pdf.length,
sha256: "",
uploadedBy: UPLOADER,
storedAt: "2026-11-09T19:00:00.000Z",
uploadId: null,
declaredSize: null,
partSize: null,
deletingAt: null,
}
async function seed(fixture: ReturnType<typeof makeDeps>, options: { optIn?: boolean } = {}) {
const sha256 = await sha256Hex(pdf)
await fixture.data.forScope(scope).of(collections).insert({ name: "contracts", createdBy: ADMIN })
if (options.optIn ?? true)
await fixture.data
.forScope(scope)
.of(autoIndex)
.insert({ collectionName: "contracts", enabledBy: ADMIN, enabledAt: "2026-11-09T18:00:00.000Z" })
const file = await fixture.data
.forScope(scope)
.of(files)
.insert({ ...fileRow, sha256 })
await fixture.store.put(scope, { key: fileRow.storageKey, body: pdf, contentType: "application/pdf" })
return { fileId: file.id, storageKey: fileRow.storageKey, contentType: "application/pdf", sha256 }
}
/** The run's document, which these cases expect there to be. */
async function indexedView<T>(view: T | null): Promise<T> {
if (view === null) throw new Error("the upload made no document")
return view
}
describe("the automatic PDF intake", () => {
it("reads an opted-in project's PDF and indexes its passages into the collection, for the administrator who opted in", async () => {
const fixture = makeDeps()
const upload = await seed(fixture)
const done = await autoIndexUpload(fixture.deps, scope, upload)
expect(done).toMatchObject({ collectionName: "contracts", fileId: upload.fileId, status: "indexed" })
expect(fixture.calls.reads).toEqual([
{ filename: "contract.pdf", idempotencyKey: `knowledge.autoindex.${upload.fileId}.${upload.sha256}` },
])
expect(fixture.calls.indexed).toEqual([{ collectionName: "contracts", fileId: upload.fileId, passages: read }])
// The job runs for nobody else: the administrator's permission is what each step leaving the process checks.
expect(fixture.calls.admitted.length).toBeGreaterThan(0)
expect(new Set(fixture.calls.admitted)).toEqual(new Set([ADMIN]))
const [row] = (await fixture.data.forScope(scope).of(documents).list({ limit: 10 })).items
expect(row).toMatchObject({ filename: "contract.pdf", requestedBy: ADMIN, status: "indexed", error: null })
// What removal needs, kept beside the document: the file's reference and the passages per page.
const [set] = (await fixture.data.forScope(scope).of(passageSets).list({ limit: 10 })).items
expect(set).toMatchObject({
collectionName: "contracts",
fileId: upload.fileId,
documentRef: `org/${scope.orgId}/${fileRow.storageKey}?${upload.fileId}`,
pages: "2,0,1",
})
})
it("is the same work once: a redelivered job reads and indexes nothing again", async () => {
const fixture = makeDeps()
const upload = await seed(fixture)
await autoIndexUpload(fixture.deps, scope, upload)
expect(await autoIndexUpload(fixture.deps, scope, upload)).toMatchObject({ status: "indexed" })
expect(fixture.calls.reads).toHaveLength(1)
expect(fixture.calls.indexed).toHaveLength(1)
})
it("reads nothing for a project that is not opted in, has knowledge off, or a file that is not a PDF", async () => {
const notOptedIn = makeDeps()
expect(await autoIndexUpload(notOptedIn.deps, scope, await seed(notOptedIn, { optIn: false }))).toBeNull()
const off = makeDeps({ enabled: false })
expect(await autoIndexUpload(off.deps, scope, await seed(off))).toBeNull()
const text = makeDeps()
const upload = await seed(text)
expect(await autoIndexUpload(text.deps, scope, { ...upload, contentType: "text/plain" })).toBeNull()
for (const fixture of [notOptedIn, off, text]) {
expect(fixture.calls.reads).toEqual([])
expect((await fixture.data.forScope(scope).of(documents).list({ limit: 10 })).items).toEqual([])
}
})
it("reads nothing once the administrator's permission is gone: the refusal is the job's", async () => {
const fixture = makeDeps()
const upload = await seed(fixture)
const refused = Object.assign(new Error("permission_missing"), { status: 403 })
const deps: KnowledgeDeps = {
...fixture.deps,
beforeIndex: async () => {
throw refused
},
}
await expect(autoIndexUpload(deps, scope, upload)).rejects.toBe(refused)
expect(fixture.calls.reads).toEqual([])
expect(fixture.calls.indexed).toEqual([])
})
it("records a PDF the reader cannot make passages of as failed, with a stable code", async () => {
const fixture = makeDeps({
reader: {
read: async () => {
throw Object.assign(new Error("block id over 256 characters"), { permanent: "not_indexable" })
},
},
})
const upload = await seed(fixture)
expect(await autoIndexUpload(fixture.deps, scope, upload)).toMatchObject({
status: "failed",
error: "not_indexable",
})
expect(fixture.calls.indexed).toEqual([])
})
it("keeps the passage layout of an index the service refused partway, so removal still tombstones it", async () => {
const fixture = makeDeps({ indexError: Object.assign(new Error("quota"), { permanent: "quota_exhausted" }) })
const upload = await seed(fixture)
const failed = await autoIndexUpload(fixture.deps, scope, upload)
expect(failed).toMatchObject({ status: "failed", error: "quota_exhausted" })
await removeDocument(fixture.deps, scope, (await indexedView(failed)).documentId)
expect(fixture.calls.removed.map((call) => call.passages.length)).toEqual([3])
expect((await fixture.data.forScope(scope).of(passageSets).list({ limit: 10 })).items).toEqual([])
})
it("reads nothing once the organization is no longer eligible or its indexing quota is spent", async () => {
const ineligible = makeDeps({
enablement: {
checkEligibility: async () => ({
ok: false,
reason: "feature_disabled",
detail: "the agent_document_tools flag is off",
}),
indexingQuota: async () => 1000,
},
})
expect(await autoIndexUpload(ineligible.deps, scope, await seed(ineligible))).toMatchObject({
status: "failed",
error: "feature_disabled",
})
const spent = makeDeps({
enablement: { checkEligibility: async () => ({ ok: true }), indexingQuota: async () => 0 },
})
expect(await autoIndexUpload(spent.deps, scope, await seed(spent))).toMatchObject({
status: "failed",
error: "quota_exhausted",
})
for (const fixture of [ineligible, spent]) {
expect(fixture.calls.reads).toEqual([])
expect(fixture.calls.indexed).toEqual([])
}
})
it("refuses to run where the enablement platform is not bound, rather than skip its checks", async () => {
const fixture = makeDeps()
const upload = await seed(fixture)
const { enablement: _unbound, ...deps } = fixture.deps
await expect(autoIndexUpload(deps, scope, upload)).rejects.toThrow("enablement")
expect(fixture.calls.reads).toEqual([])
})
it("does not read a file whose bytes changed since the upload", async () => {
const fixture = makeDeps()
const upload = await seed(fixture)
expect(await autoIndexUpload(fixture.deps, scope, { ...upload, sha256: "0".repeat(64) })).toBeNull()
expect(fixture.calls.reads).toEqual([])
})
it("tombstones every passage it indexed when the document is removed, even after the file is gone", async () => {
const fixture = makeDeps()
const upload = await seed(fixture)
const done = await autoIndexUpload(fixture.deps, scope, upload)
await fixture.data.forScope(scope).of(files).delete(upload.fileId)
await removeDocument(fixture.deps, scope, (await indexedView(done)).documentId)
expect(fixture.calls.removed).toEqual([
{
documentRef: `org/${scope.orgId}/${fileRow.storageKey}?${upload.fileId}`,
passages: [
{ page: 1, index: 0 },
{ page: 1, index: 1 },
{ page: 3, index: 0 },
],
},
])
expect((await fixture.data.forScope(scope).of(documents).list({ limit: 10 })).items).toEqual([])
expect((await fixture.data.forScope(scope).of(passageSets).list({ limit: 10 })).items).toEqual([])
})
})
describe("the intake's place in the manifest", () => {
it("contributes a files processor for PDFs whose job the module runs", () => {
const contributions = (knowledgeModule.extends ?? []).filter((entry) => entry.point.id === filesProcessors.id)
expect(contributions.flatMap((entry) => entry.values)).toEqual([
{ key: AUTO_INDEX_PROCESSOR_KEY, contentTypes: ["application/pdf"], job: AUTO_INDEX_JOB },
])
expect(Object.keys(knowledgeModule.jobs ?? {})).toContain(AUTO_INDEX_JOB)
})
it("declares the platform operations the job uses: the knowledge service's and the document reader's", () => {
const operations = (service: string) =>
knowledgeModule.serviceRequirements?.find((requirement) => requirement.service === service)?.operations
expect(operations("knowledge")?.[AUTO_INDEX_JOB]).toEqual(["knowledge.collections:use"])
expect(operations("ocr")?.[AUTO_INDEX_JOB]).toEqual(["ocr:use"])
})
})
@@ -0,0 +1,73 @@
// A project's automatic PDF indexing (template lesson core-11): an administrator turns it on into an
// existing collection, and is recorded as the one whose permission each indexing job re-checks; off
// is no row. Another project's collection is not found, and a project with knowledge off refuses.
import type { TenantScope } from "@conusai-kit/core"
import { createMemoryDataSource } from "@conusai-kit/data"
import { DEFAULT_FILES_POLICY, files } from "@conusai-kit/files"
import { fakeObjectStore } from "@conusai-kit/testing"
import { describe, expect, it } from "vitest"
import { autoIndex, collections, KNOWLEDGE_ENTITIES } from "../contract/index.js"
import { getAutoIndex, setAutoIndex } from "../core/auto-index.js"
import type { KnowledgeDeps } from "../core/ports.js"
const ADMIN = "01900000-0000-7000-0000-000000000009"
const scope: TenantScope = {
level: "project",
actor: { userId: ADMIN },
orgId: "01900000-0000-7000-0000-000000000001",
workspaceId: "01900000-0000-7000-0000-000000000002",
projectId: "01900000-0000-7000-0000-000000000003",
}
const otherProject: TenantScope = { ...scope, projectId: "01900000-0000-7000-0000-000000000004" }
function makeDeps(enabled = true) {
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES, files] })
const deps: KnowledgeDeps & { now: () => number } = {
data,
files: { data, store: fakeObjectStore(), policy: DEFAULT_FILES_POLICY },
flags: { enabled: async () => enabled },
now: () => Date.parse("2026-11-09T19:00:00.000Z"),
}
return { data, deps }
}
describe("automatic PDF indexing of a project", () => {
it("is off until an administrator names a collection, and records who turned it on", async () => {
const { data, deps } = makeDeps()
await data.forScope(scope).of(collections).insert({ name: "contracts", createdBy: ADMIN })
expect(await getAutoIndex(deps, scope)).toEqual({ collectionName: null })
expect(await setAutoIndex(deps, scope, "contracts")).toEqual({ collectionName: "contracts" })
expect(await getAutoIndex(deps, scope)).toEqual({ collectionName: "contracts" })
const [row] = (await data.forScope(scope).of(autoIndex).list({ limit: 10 })).items
expect(row).toMatchObject({ collectionName: "contracts", enabledBy: ADMIN, enabledAt: "2026-11-09T19:00:00.000Z" })
// Another project of the same organization is not opted in.
expect(await getAutoIndex(deps, otherProject)).toEqual({ collectionName: null })
})
it("keeps one row per project: naming another collection moves it, and off removes it", async () => {
const { data, deps } = makeDeps()
for (const name of ["contracts", "invoices"])
await data.forScope(scope).of(collections).insert({ name, createdBy: ADMIN })
await setAutoIndex(deps, scope, "contracts")
await setAutoIndex(deps, scope, "invoices")
expect(
(await data.forScope(scope).of(autoIndex).list({ limit: 10 })).items.map((row) => row.collectionName),
).toEqual(["invoices"])
expect(await setAutoIndex(deps, scope, null)).toEqual({ collectionName: null })
expect((await data.forScope(scope).of(autoIndex).list({ limit: 10 })).items).toEqual([])
// Off when already off writes nothing and answers the same.
expect(await setAutoIndex(deps, scope, null)).toEqual({ collectionName: null })
})
it("answers another project's collection as not found, and refuses while knowledge is off", async () => {
const { data, deps } = makeDeps()
await data.forScope(otherProject).of(collections).insert({ name: "theirs", createdBy: ADMIN })
await expect(setAutoIndex(deps, scope, "theirs")).rejects.toMatchObject({ status: 404 })
const off = makeDeps(false)
await off.data.forScope(scope).of(collections).insert({ name: "contracts", createdBy: ADMIN })
await expect(setAutoIndex(off.deps, scope, "contracts")).rejects.toMatchObject({ status: 403 })
expect((await off.data.forScope(scope).of(autoIndex).list({ limit: 10 })).items).toEqual([])
})
})
+134
View File
@@ -0,0 +1,134 @@
// SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0
import {
buildRegistry,
Container,
contribute, // kit:guides
type ModuleManifest,
type TenantScope,
} from "@conusai-kit/core"
import { createMemoryDataSource, DATA_SOURCE, entitiesOf, JOB_QUEUE, OBJECT_STORE } from "@conusai-kit/data"
import * as guidesKit from "@conusai-kit/guides" // kit:guides
import { projects, workspaces } from "@conusai-kit/tenancy"
import { fakeObjectStore } from "@conusai-kit/testing"
import { expect, it } from "vitest"
import { custodyModule } from "../../custody.js"
import { OPTIONAL_MODULE_IDS } from "../../inclusion.js"
import { appModules } from "../../index.js"
import { documents, KNOWLEDGE_ENTITIES, KNOWLEDGE_SCOPE, KNOWLEDGE_SEARCH, knowledgeModule } from "../index.js"
const actor = { userId: "01900000-0000-7000-0000-000000000009" }
const orgId = "01900000-0000-7000-0000-000000000001"
/**
* A module as the app composes it without the opt-in guides module: its contributions to the guides extension
* points go with it, whichever module made them (an app's own module may bring code guides too).
*/
function withoutGuideContributions(entry: ModuleManifest): ModuleManifest {
return entry.extends
? { ...entry, extends: entry.extends.filter((item) => !item.point.id.startsWith("guides.")) }
: entry
}
// kit:guides:begin
it("drops guides contributions from every module, not only notes", () => {
// An app's own module with a code guide: knowledge stands in for it.
const guided: ModuleManifest = {
...knowledgeModule,
extends: [...(knowledgeModule.extends ?? []), contribute(guidesKit.guidesCode, [])],
}
expect(withoutGuideContributions(guided).extends?.filter((item) => item.point.id.startsWith("guides."))).toEqual([])
})
// kit:guides:end
it("off retains the original Knowledge rows/schema and executes only source cleanup", async () => {
const base = appModules.filter(
(entry) => !OPTIONAL_MODULE_IDS.some((id) => id === entry.id) && entry.id !== "assistant",
)
const independent = base.map(withoutGuideContributions)
const active = buildRegistry([...independent, knowledgeModule])
const retained = buildRegistry([...independent, custodyModule(knowledgeModule)])
expect(retained.migrations.postgres).toEqual(active.migrations.postgres)
expect(entitiesOf(retained, "postgres").map((entity) => entity.table)).toEqual(
entitiesOf(active, "postgres").map((entity) => entity.table),
)
expect([...retained.procedures.keys()].filter((key) => key.startsWith("knowledge."))).toEqual([])
expect(Object.keys(retained.jobs).filter((key) => key.startsWith("knowledge."))).toEqual(["knowledge.remove"])
// The project the deleted file was in, as the organization has it: a delivery acts only in a project that exists.
const data = createMemoryDataSource({ entities: [workspaces, projects, ...KNOWLEDGE_ENTITIES] })
const workspace = await data
.forScope({ level: "org", actor, orgId })
.of(workspaces)
.insert({ slug: "main", name: "Main", isDefault: true, visibility: "org" })
const project = await data
.forScope({ level: "workspace", actor, orgId, workspaceId: workspace.id })
.of(projects)
.insert({ slug: "site", name: "Site", moduleKey: "notes", archivedAt: null })
const scope: TenantScope = { level: "project", actor, orgId, workspaceId: workspace.id, projectId: project.id }
const row = await data.forScope(scope).of(documents).insert({
collectionName: "synthetic",
fileId: "01900000-0000-7000-0000-000000000004",
filename: "synthetic.txt",
status: "indexed",
error: null,
indexedAt: "2026-11-03T00:00:00Z",
requestedBy: "synthetic",
})
const module = custodyModule(knowledgeModule)
expect(await data.forScope(scope).of(documents).get(row.id)).toEqual(row)
const container = new Container()
container.value(DATA_SOURCE, data)
container.value(OBJECT_STORE, fakeObjectStore())
container.value(KNOWLEDGE_SCOPE, scope)
const enqueued: string[] = []
container.value(JOB_QUEUE, {
enqueue: async (_scope, job) => {
enqueued.push(job)
return "synthetic-job"
},
})
const deletion = module.events?.subscribes?.find((entry) => entry.event.name === "files.FileDeleted")
if (!deletion) throw new Error("source cleanup subscription missing")
await deletion.handler(
{
name: "files.FileDeleted",
// As kit-files publishes it: the deleted file's storage key comes with the record.
payload: {
orgId: scope.orgId,
workspaceId: scope.workspaceId,
projectId: scope.projectId,
fileId: row.fileId,
storageKey: "synthetic/key",
},
},
{ container } as never,
)
expect((await data.forScope(scope).of(documents).get(row.id))?.status).toBe("cancelled")
const event = (await data.outbox.pending(10))[0]
if (!event) throw new Error("cleanup event missing")
const cleanup = module.events?.subscribes?.find((entry) => entry.event.name === "knowledge.documentRemoveRequested")
if (!cleanup) throw new Error("cleanup subscription missing")
await cleanup.handler({ name: event.name, payload: event.payload }, { container } as never)
expect(enqueued).toEqual(["knowledge.remove"])
expect(module.events?.subscribes?.some((entry) => entry.event.name === "knowledge.documentIndexRequested")).toBe(
false,
)
const removed: string[] = []
container.value(KNOWLEDGE_SEARCH, {
remove: async (_scope: unknown, input: { fileId: string }) => {
removed.push(input.fileId)
},
index: async () => {
throw new Error("processing must be fenced")
},
search: async () => {
throw new Error("interactive search must be fenced")
},
} as never)
// The removal job resolves the original repositories and ports; no alternate store is created.
const remove = module.jobs?.["knowledge.remove"]
if (!remove) throw new Error("cleanup job missing")
await remove.handler({ documentId: row.id, collectionName: row.collectionName, fileId: row.fileId }, {
container,
} as never)
expect(removed).toEqual([row.fileId])
expect(await data.forScope(scope).of(documents).get(row.id)).toBeNull()
})
@@ -0,0 +1,78 @@
// SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0
// A delivery's project comes from the record, inside the delivery's organization (adapters/jobs.ts). The record's
// workspace and project must exist in that organization before knowledge acts on them; one that names another
// workspace's project, or one the organization does not have, is refused by name and nothing is enqueued.
import { Container, type EventContext, newId, type OrgScope } from "@conusai-kit/core"
import { createMemoryDataSource, DATA_SOURCE, JOB_QUEUE } from "@conusai-kit/data"
import { projects, workspaces } from "@conusai-kit/tenancy"
import { fakeJobQueue } from "@conusai-kit/testing"
import { describe, expect, it } from "vitest"
import { KNOWLEDGE_SUBSCRIPTIONS } from "../adapters/jobs.js"
import { KNOWLEDGE_ENTITIES, KNOWLEDGE_SCOPE } from "../index.js"
async function delivery(archivedAt: string | null = null) {
const data = createMemoryDataSource({ entities: [workspaces, projects, ...KNOWLEDGE_ENTITIES] })
const orgId = newId()
const org: OrgScope = { level: "org", actor: { userId: newId() }, orgId }
const workspace = await data
.forScope(org)
.of(workspaces)
.insert({ slug: "main", name: "Main", isDefault: true, visibility: "org" })
const other = await data
.forScope(org)
.of(workspaces)
.insert({ slug: "other", name: "Other", isDefault: false, visibility: "org" })
const project = await data
.forScope({ level: "workspace", actor: org.actor, orgId, workspaceId: workspace.id })
.of(projects)
.insert({ slug: "site", name: "Site", moduleKey: "notes", archivedAt })
const queue = fakeJobQueue()
const container = new Container()
container.value(DATA_SOURCE, data)
container.value(JOB_QUEUE, queue)
// The relay's delivery: the organization's scope, with a system actor.
container.value(KNOWLEDGE_SCOPE, org)
const ctx = { container } as unknown as EventContext
return { ctx, queue, orgId, workspaceId: workspace.id, otherWorkspaceId: other.id, projectId: project.id }
}
const indexRequested = KNOWLEDGE_SUBSCRIPTIONS[1]
const request = (ids: { orgId: string; workspaceId: string; projectId: string }) => ({
name: "knowledge.DocumentIndexRequested",
payload: { ...ids, documentId: newId(), collectionName: "contracts", fileId: newId() },
})
describe("a knowledge delivery acts only in a project of its organization", () => {
it("enqueues the job in the record's project when it exists there", async () => {
const { ctx, queue, orgId, workspaceId, projectId } = await delivery()
await indexRequested?.handler(request({ orgId, workspaceId, projectId }) as never, ctx)
expect(queue.of("knowledge.index")).toHaveLength(1)
})
it("refuses, by name, a workspace the organization does not have", async () => {
const { ctx, queue, orgId, projectId } = await delivery()
const workspaceId = newId()
await expect(indexRequested?.handler(request({ orgId, workspaceId, projectId }) as never, ctx)).rejects.toThrow(
new RegExp(`workspace ${workspaceId}`),
)
expect(queue.of("knowledge.index")).toEqual([])
})
it("refuses, by name, a project that is not in the record's workspace", async () => {
const { ctx, queue, orgId, otherWorkspaceId, projectId } = await delivery()
await expect(
indexRequested?.handler(request({ orgId, workspaceId: otherWorkspaceId, projectId }) as never, ctx),
).rejects.toThrow(new RegExp(`project ${projectId}`))
expect(queue.of("knowledge.index")).toEqual([])
})
it("acts on a record of an archived project: an archive hides a project and does not end its work", async () => {
// The kit's one rule (kit-tenancy suspension.ts): an archived project refuses a person's write, not work with
// no person behind it. A deleted file's text must still leave the index, and an index request a member made
// before the archive still completes.
const { ctx, queue, orgId, workspaceId, projectId } = await delivery(new Date("2026-11-09T00:00:00Z").toISOString())
await indexRequested?.handler(request({ orgId, workspaceId, projectId }) as never, ctx)
expect(queue.of("knowledge.index")).toHaveLength(1)
})
})
@@ -0,0 +1,349 @@
// The document requests and the index job's orchestration: idempotent (collection, file) pairs,
// the flag gate, the content-type and size admission, the row's progress states, and which
// failures are permanent.
import type { TenantScope } from "@conusai-kit/core"
import { createMemoryDataSource } from "@conusai-kit/data"
import { DEFAULT_FILES_POLICY, files, sha256Hex } from "@conusai-kit/files"
import { fakeObjectStore } from "@conusai-kit/testing"
import { describe, expect, it } from "vitest"
import { collections, documents, KNOWLEDGE_ENTITIES } from "../contract/index.js"
import { listDocuments, requestIndex, requestRemove, requestSourceRemoval } from "../core/documents.js"
import { indexDocument, removeDocument } from "../core/index-job.js"
import type { KnowledgeDeps } from "../core/ports.js"
import { searchKnowledge } from "../core/search.js"
const USER = "01900000-0000-7000-0000-000000000009"
const scope: TenantScope = {
level: "project",
actor: { userId: USER },
orgId: "01900000-0000-7000-0000-000000000001",
workspaceId: "01900000-0000-7000-0000-000000000002",
projectId: "01900000-0000-7000-0000-000000000003",
}
const fileRow = {
status: "stored" as const,
storageKey: "k/seed",
filename: "a.txt",
contentType: "text/plain",
size: 12,
sha256: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
uploadedBy: USER,
storedAt: "2026-10-23T00:00:00.000Z",
uploadId: null,
declaredSize: null,
partSize: null,
deletingAt: null,
}
const makeDeps = (options: { flag?: boolean; search?: KnowledgeDeps["search"] } = {}) => {
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES, files] })
const store = fakeObjectStore()
const search = options.search ?? {
readiness: async () => ({ state: "ready" as const, reasons: [] }),
index: async () => undefined,
search: async () => [],
remove: async () => undefined,
}
const deps: KnowledgeDeps = {
data,
files: { data, store, policy: DEFAULT_FILES_POLICY },
search,
flags: { enabled: async () => options.flag ?? true },
}
void deps
return { data, deps, store }
}
const seed = async (data: ReturnType<typeof createMemoryDataSource>, contentType = "text/plain") => {
await data.forScope(scope).of(collections).insert({ name: "contracts", createdBy: USER })
const row = await data
.forScope(scope)
.of(files)
.insert({ ...fileRow, ...(contentType === "text/plain" ? {} : { contentType }) })
return row.id
}
/** Seeds the file row AND the object the row names, the way a completed upload leaves both. */
const seedWithBytes = async (
data: ReturnType<typeof createMemoryDataSource>,
store: ReturnType<typeof fakeObjectStore>,
bytes: Uint8Array,
) => {
const fileId = await seed(data)
await data
.forScope(scope)
.of(files)
.update(fileId, { size: bytes.length, sha256: await sha256Hex(bytes) }, 1)
await store.put(scope, { key: fileRow.storageKey, body: bytes, contentType: "text/plain" })
return fileId
}
describe("requesting a document's indexing", () => {
it("creates the row, publishes the event, and a repeat request changes nothing", async () => {
const { data, deps } = makeDeps()
const fileId = await seed(data)
const first = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
expect(first.status).toBe("queued")
const second = await requestIndex(deps, scope, { collectionName: "contracts", fileId: first.fileId })
expect(second.documentId).toBe(first.documentId)
expect(await listDocuments(deps, scope, "contracts")).toEqual({
documents: [expect.objectContaining({ documentId: first.documentId, status: "queued" })],
})
// One outbox event: the subscriber's enqueue is the exactly-once path.
const outbox = data.outbox.pending(10)
expect((await outbox).filter((record) => record.name === "knowledge.documentIndexRequested")).toHaveLength(1)
})
it("re-queues an indexed pair instead of duplicating it", async () => {
const { data, deps } = makeDeps()
const fileId = await seed(data)
const first = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
await data.forScope(scope).of(documents).update(first.documentId, { status: "indexed" }, 1)
const again = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
expect(again.documentId).toBe(first.documentId)
expect(again.status).toBe("queued")
})
it("refuses when the flag is off, and admits text files only, within the size cap", async () => {
const { data, deps } = makeDeps({ flag: false })
const fileId = await seed(data)
const off = await requestIndex(deps, scope, { collectionName: "contracts", fileId }).catch((e: unknown) => e)
expect((off as { status?: number }).status).toBe(403)
const refusing = makeDeps()
const pdfFile = await seed(refusing.data, "application/pdf")
const pdf = await requestIndex(refusing.deps, scope, { collectionName: "contracts", fileId: pdfFile }).catch(
(e: unknown) => e,
)
expect((pdf as { status?: number }).status).toBe(422)
const big = makeDeps()
const bigFile = await seed(big.data)
await big.data.forScope(scope).of(files).update(bigFile, { size: 2_000_000 }, 1)
const tooBig = await requestIndex(big.deps, scope, { collectionName: "contracts", fileId: bigFile }).catch(
(e: unknown) => e,
)
expect((tooBig as { status?: number }).status).toBe(422)
})
})
describe("current retrieval authority", () => {
it("withholds cancelled and unindexed documents even while remote deletion retries", async () => {
const { data, deps } = makeDeps()
const fileId = await seed(data)
const row = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
const searching = {
...deps,
search: {
...deps.search,
readiness: async () => ({ state: "ready" as const, reasons: [] }),
index: async () => undefined,
remove: async () => undefined,
search: async () => [
{ fileId, filename: "a.txt", collectionName: "contracts", snippet: "private source", score: 1, chunk: 1 },
],
},
}
expect((await searchKnowledge(searching, scope, { collectionName: "contracts", query: "source" })).results).toEqual(
[],
)
await data.forScope(scope).of(documents).update(row.documentId, { status: "indexed" }, 1)
expect(
(await searchKnowledge(searching, scope, { collectionName: "contracts", query: "source" })).results,
).toHaveLength(1)
await requestRemove(deps, scope, { collectionName: "contracts", fileId })
expect((await searchKnowledge(searching, scope, { collectionName: "contracts", query: "source" })).results).toEqual(
[],
)
// A removal a person asks for names the stored file's object, read when it is asked for: if the file is deleted
// before the job runs, the job still removes the right text (F407, review R1).
const asked = (await data.outbox.pending(100)).filter((event) => event.name === "knowledge.documentRemoveRequested")
expect(asked).toHaveLength(1)
expect(asked[0]?.payload).toMatchObject({ storageKey: "k/seed" })
})
it("refuses a person's removal of a file being deleted, which leaves the document to the deletion's own removal", async () => {
const { data, deps } = makeDeps()
const fileId = await seed(data)
const row = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
await data.forScope(scope).of(documents).update(row.documentId, { status: "indexed" }, 1)
const file = await data.forScope(scope).of(files).require(fileId)
await data
.forScope(scope)
.of(files)
.update(fileId, { status: "deleting", deletingAt: new Date().toISOString() }, file.version)
await expect(requestRemove(deps, scope, { collectionName: "contracts", fileId })).rejects.toMatchObject({
status: 409,
})
// Still active, so `FileDeleted` (which carries the key) cancels it and asks for the removal.
expect((await data.forScope(scope).of(documents).require(row.documentId)).status).toBe("indexed")
expect(
(await data.outbox.pending(100)).filter((event) => event.name === "knowledge.documentRemoveRequested"),
).toEqual([])
})
})
describe("retained remote citations after source deletion", () => {
it("withholds a deleting or missing file even when its index row still says indexed", async () => {
const { data, deps } = makeDeps()
const fileId = await seed(data)
const row = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
await data.forScope(scope).of(documents).update(row.documentId, { status: "indexed" }, 1)
if (!deps.search) throw new Error("the retained citation fixture must configure search")
const searching = {
...deps,
search: {
...deps.search,
search: async () => [
{ fileId, filename: "a.txt", collectionName: "contracts", snippet: "retained source", score: 1, chunk: 1 },
],
},
}
expect(
(await searchKnowledge(searching, scope, { collectionName: "contracts", query: "source" })).results,
).toHaveLength(1)
await data.forScope(scope).of(files).update(fileId, { status: "deleting" }, 1)
expect((await searchKnowledge(searching, scope, { collectionName: "contracts", query: "source" })).results).toEqual(
[],
)
const { storageKey } = (await data.forScope(scope).of(files).get(fileId)) ?? { storageKey: "" }
await data.forScope(scope).of(files).delete(fileId)
await requestSourceRemoval({ ...deps, flags: { enabled: () => false } }, scope, fileId, storageKey)
await requestSourceRemoval(deps, scope, fileId, storageKey)
expect((await listDocuments(deps, scope, "contracts")).documents[0]?.status).toBe("cancelled")
const cleanup = (await data.outbox.pending(100)).filter(
(event) => event.name === "knowledge.documentRemoveRequested",
)
expect(cleanup).toHaveLength(1)
// The deleted file's key travels with the request: its row is gone, so the removal cannot look it up (F407).
expect(cleanup[0]?.payload).toMatchObject({ storageKey })
expect((await searchKnowledge(searching, scope, { collectionName: "contracts", query: "source" })).results).toEqual(
[],
)
})
})
describe("the index job's orchestration", () => {
it("moves the row queued -> indexing -> indexed and calls the search service once", async () => {
const indexed: string[] = []
const { data, deps, store } = makeDeps({
search: {
readiness: async () => ({ state: "ready", reasons: [] }),
index: async (_scope, input) => {
indexed.push(input.fileId)
},
search: async () => [],
remove: async () => undefined,
},
})
const fileId = await seedWithBytes(data, store, new TextEncoder().encode("hello knowledge"))
const requested = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
const done = await indexDocument(deps, scope, requested.documentId)
expect(done.status).toBe("indexed")
expect(indexed).toEqual([fileId])
// A redelivered job finds the row indexed and does the external work again never.
const again = await indexDocument(deps, scope, requested.documentId)
expect(again.status).toBe("indexed")
expect(indexed).toEqual([fileId])
})
it("records a permanent failure on the row; a transient one throws for the queue to retry", async () => {
const permanent = makeDeps({
search: {
readiness: async () => ({ state: "ready", reasons: [] }),
index: async () => {
throw Object.assign(new Error("not text"), { permanent: "not_text" })
},
search: async () => [],
remove: async () => undefined,
},
})
const permanentFile = await seedWithBytes(
permanent.data,
permanent.store,
new TextEncoder().encode("not text at all"),
)
const requested = await requestIndex(permanent.deps, scope, { collectionName: "contracts", fileId: permanentFile })
const failed = await indexDocument(permanent.deps, scope, requested.documentId)
expect(failed.status).toBe("failed")
expect(failed.error).toBe("not_text")
const transient = makeDeps({
search: {
readiness: async () => ({ state: "ready", reasons: [] }),
index: async () => {
throw new Error("the store hiccuped")
},
search: async () => [],
remove: async () => undefined,
},
})
const transientFile = await seedWithBytes(transient.data, transient.store, new TextEncoder().encode("hello"))
const requested2 = await requestIndex(transient.deps, scope, { collectionName: "contracts", fileId: transientFile })
await expect(indexDocument(transient.deps, scope, requested2.documentId)).rejects.toThrow("hiccuped")
let resumed = 0
const retry = {
...transient.deps,
search: {
...transient.deps.search,
readiness: async () => ({ state: "ready" as const, reasons: [] }),
search: async () => [],
remove: async () => undefined,
index: async () => {
resumed++
},
},
}
expect((await indexDocument(retry, scope, requested2.documentId)).status).toBe("indexed")
expect(resumed).toBe(1)
})
it("removal deletes the row and asks the service to drop the vectors", async () => {
const removed: string[] = []
const { data, deps } = makeDeps({
search: {
readiness: async () => ({ state: "ready", reasons: [] }),
index: async () => undefined,
search: async () => [],
remove: async (_scope, input) => {
removed.push(input.fileId)
},
},
})
const fileId = await seed(data)
const requested = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
const cancelled = await requestRemove(deps, scope, { collectionName: "contracts", fileId })
expect(cancelled.status).toBe("cancelled")
expect((await indexDocument(deps, scope, requested.documentId)).status).toBe("cancelled")
// The remove job's body: delete the vectors, then the row.
await removeDocument(deps, scope, requested.documentId)
expect(removed).toEqual([fileId])
expect((await listDocuments(deps, scope, "contracts")).documents).toEqual([])
})
})
it("persists a final transient indexing failure but never overwrites cancellation on a lost claim", async () => {
const transient = makeDeps({
search: {
readiness: async () => ({ state: "ready", reasons: [] }),
index: async () => {
throw new Error("temporary provider outage")
},
search: async () => [],
remove: async () => {},
},
})
const fileId = await seedWithBytes(transient.data, transient.store, new TextEncoder().encode("synthetic source"))
const requested = await requestIndex(transient.deps, scope, { collectionName: "contracts", fileId })
const outcome = await indexDocument(transient.deps, scope, requested.documentId, { finalAttempt: true })
expect(outcome).toMatchObject({ status: "failed", error: "processing_exhausted" })
const controller = new AbortController()
controller.abort()
const next = await requestIndex(transient.deps, scope, { collectionName: "contracts", fileId })
await expect(
indexDocument(transient.deps, scope, next.documentId, { finalAttempt: true, signal: controller.signal }),
).rejects.toBeDefined()
expect((await transient.data.forScope(scope).of(documents).get(next.documentId))?.status).not.toBe("failed")
})
+30
View File
@@ -0,0 +1,30 @@
// The module's messages: en and lt declare the same keys, or a page in one language breaks.
import { buildRegistry } from "@conusai-kit/core"
import { filesModule } from "@conusai-kit/files"
import { createI18n } from "@conusai-kit/i18n"
import { tenancyModule } from "@conusai-kit/tenancy"
import { describe, expect, it } from "vitest"
import { knowledgeModule } from "../index.js"
describe("the knowledge module's messages", () => {
const catalogues = buildRegistry([tenancyModule, filesModule, knowledgeModule]).catalogues
it("declares en and lt, and every en key resolves in lt", () => {
const en = catalogues.en?.knowledge ?? {}
const lt = catalogues.lt?.knowledge ?? {}
const flatten = (tree: Record<string, unknown>, prefix = ""): string[] =>
Object.entries(tree).flatMap(([key, value]) =>
typeof value === "string" ? [`${prefix}${key}`] : flatten(value as Record<string, unknown>, `${prefix}${key}.`),
)
const enKeys = flatten(en)
const ltKeys = new Set(flatten(lt))
expect(enKeys.length).toBeGreaterThan(0)
for (const key of enKeys) expect(ltKeys.has(key), `knowledge.${key} has no Lithuanian message`).toBe(true)
})
it("formats a message with values in both locales", () => {
const i18n = createI18n("lt", { lt: { knowledge: catalogues.lt?.knowledge ?? {} } })
expect(i18n.message("knowledge.status.documents", { queued: 1, indexing: 2, indexed: 3, failed: 0 })).toContain("1")
})
})
+59
View File
@@ -0,0 +1,59 @@
// What the generated tenant-isolation suite (test/isolation.test.ts) needs of this module: rows it
// seeds in every tenant, and one fixture per tenant-scoped procedure. A fixture is built from the
// OTHER tenant's ids: the call is made as a member of organization A with organization B's slugs
// and ids, and has to be refused or answered empty.
import type { TransportCaller } from "@conusai-kit/core"
import { type IsolationFixture, type IsolationTenant, uploadViaProcedures } from "@conusai-kit/testing"
/** What a module's isolation file exports: the shape the suite reads. */
export interface ModuleIsolation {
seed(
call: TransportCaller,
inProject: { org: string; workspace: string; project: string },
): Promise<Record<string, string>>
fixtures: Record<string, IsolationFixture>
}
const NOWHERE = "00000000-0000-0000-0000-000000000000"
const inProjectOf = (t: IsolationTenant) => ({ org: t.orgSlug, workspace: t.workspaceSlug, project: t.projectSlug })
const collectionOf = (t: IsolationTenant) => t.extra.knowledgeCollection ?? "contracts"
const fileOf = (t: IsolationTenant) => t.extra.knowledgeFileId ?? NOWHERE
/** A knowledge_document row the suite seeds in each tenant (test/isolation.test.ts): indexing needs the module on. */
const documentOf = (t: IsolationTenant) => t.extra.knowledgeDocumentId ?? NOWHERE
export const isolation: ModuleIsolation = {
async seed(call, inProject) {
const file = await uploadViaProcedures(call, inProject, {
filename: "knowledge-seed.txt",
contentType: "text/plain",
bytes: new TextEncoder().encode("seed knowledge"),
})
// The seed needs a collection row to exist; collection.create needs the admin role, which the
// isolation harness does not grant the seeding caller, so the fixture names a collection and
// the write fixtures create it through the manage permission where a denial is the assertion.
return { knowledgeCollection: "contracts", knowledgeFileId: file.fileId }
},
fixtures: {
"knowledge.collection.list": { input: (t) => inProjectOf(t) },
"knowledge.collection.create": { input: (t) => ({ ...inProjectOf(t), name: "intruders" }) },
"knowledge.document.list": {
input: (t) => ({ ...inProjectOf(t), collectionName: collectionOf(t) }),
},
// The live collection's read-back: another tenant's document id answers nothing.
"knowledge.document.getMany": { input: (t) => ({ ...inProjectOf(t), ids: [documentOf(t)] }) },
"knowledge.document.index": {
input: (t) => ({ ...inProjectOf(t), collectionName: collectionOf(t), fileId: fileOf(t) }),
},
"knowledge.document.remove": {
input: (t) => ({ ...inProjectOf(t), collectionName: collectionOf(t), fileId: fileOf(t) }),
},
"knowledge.search": { input: (t) => ({ ...inProjectOf(t), collectionName: collectionOf(t), query: "seed" }) },
"knowledge.status": { input: (t) => inProjectOf(t) },
"knowledge.settings.get": { input: (t) => ({ org: t.orgSlug }) },
"knowledge.settings.enableDocumentAi": { input: (t) => ({ org: t.orgSlug }) },
"knowledge.settings.disableDocumentAi": { input: (t) => ({ org: t.orgSlug }) },
"knowledge.autoindex.get": { input: (t) => inProjectOf(t) },
"knowledge.autoindex.set": { input: (t) => ({ ...inProjectOf(t), collectionName: collectionOf(t) }) },
},
}
@@ -0,0 +1,56 @@
// The module's migrations are its own files, each exactly `schemaStatements()` over the entities it adds
// (decision AT-D20: the entity definitions are the one source of a column). The module ships the
// Postgres profile; the Convex binding is provided for the integration owner and has no SQL.
import { readFileSync } from "node:fs"
import { schemaStatements } from "@conusai-kit/data-kysely"
import { describe, expect, it } from "vitest"
import { autoIndex, KNOWLEDGE_ENTITIES, KNOWLEDGE_ENTITIES_V1, knowledgeModule, passageSets } from "../index.js"
/** The statements for these tables alone: `schemaStatements` leads with the kernel's own two. */
function tablesOf(entities: Parameters<typeof schemaStatements>[0]): string {
const kernel = schemaStatements([]).length
return `${schemaStatements(entities)
.slice(kernel)
.map((statement) => `${statement};`)
.join("\n\n")}\n`
}
describe("the migration of the knowledge module", () => {
it("is its own file, exactly its three tables as first shipped", async () => {
await expect(
`-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0\n${tablesOf(KNOWLEDGE_ENTITIES_V1)}`,
).toMatchFileSnapshot("../migrations/0001_knowledge.sql")
})
it("adds the auto-index opt-in in the second file, exactly its table", async () => {
await expect(
`-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0\n${tablesOf([autoIndex])}`,
).toMatchFileSnapshot("../migrations/0002_knowledge_auto_index.sql")
})
it("adds the PDF intake's passage record in the third file, exactly its table", async () => {
await expect(
`-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0\n${tablesOf([passageSets])}`,
).toMatchFileSnapshot("../migrations/0003_knowledge_passage_set.sql")
expect(KNOWLEDGE_ENTITIES).toEqual([...KNOWLEDGE_ENTITIES_V1, autoIndex, passageSets])
})
it("is declared by the manifest, so the migrate job finds it, and every table is row-level secured", () => {
expect(knowledgeModule.migrations).toEqual({ postgres: "migrations" })
expect(knowledgeModule.profiles).toEqual(["postgres"])
const sql = ["0001_knowledge.sql", "0002_knowledge_auto_index.sql", "0003_knowledge_passage_set.sql"]
.map((file) => readFileSync(new URL(`../migrations/${file}`, import.meta.url), "utf8"))
.join("\n")
for (const table of [
"knowledge_collection",
"knowledge_document",
"knowledge_config",
"knowledge_auto_index",
"knowledge_passage_set",
]) {
expect(sql).toContain(`create table "${table}"`)
expect(sql).toContain(`alter table "${table}" force row level security`)
}
})
})
+208
View File
@@ -0,0 +1,208 @@
// The knowledge page on F10: its loader reads through the caller and returns plain data with every label
// translated, and its view renders what reached the browser. The service's health, the indexing indicator
// and the search stay three separate things: "indexing" is not "down".
import { AppError, buildRegistry, Container, notFound, type TransportCaller } from "@conusai-kit/core"
import { builtInCatalogues, createI18n, uiLabels } from "@conusai-kit/i18n"
import { createDataClient, DataProvider } from "@conusai-kit/ui/data"
import { UiLabelsProvider } from "@conusai-kit/ui/primitives"
import { renderToStaticMarkup as renderMarkup } from "react-dom/server"
import { describe, expect, it, vi } from "vitest"
import { appModules } from "../../index.js"
import { modulePages } from "../../pages.js"
import { knowledgePage } from "../ui/pages.js"
const kitStrings = uiLabels(createI18n("en", builtInCatalogues))
const renderToStaticMarkup = (node: React.ReactNode) =>
renderMarkup(<UiLabelsProvider labels={kitStrings}>{node}</UiLabelsProvider>)
const catalogues = buildRegistry([...appModules]).catalogues
const i18n = createI18n("en", {
en: { knowledge: catalogues.en?.knowledge ?? {}, app: { failed: "That did not work: {detail}" } },
})
const slugs = { org: "acme", workspace: "ws", project: "p" }
const callerOf = (answers: Record<string, unknown>, asked: unknown[] = []): TransportCaller =>
({
call: async (contract: { name: string }, input: unknown) => {
asked.push([contract.name, input])
const answer = answers[contract.name]
if (answer instanceof Error) throw answer
return answer
},
}) as unknown as TransportCaller
const load = (segments: string[], answers: Record<string, unknown>, asked?: unknown[]) =>
knowledgePage.load({
slugs,
segments,
search: {},
caller: callerOf(answers, asked),
i18n,
container: new Container(),
})
/** The page as the shell renders it: the loader's data through JSON, as the browser receives it, then the view. */
const render = async (answers: Record<string, unknown>) => {
const data = await load([], answers)
expect(JSON.parse(JSON.stringify(data))).toStrictEqual(data)
const { View } = knowledgePage
return renderToStaticMarkup(
<DataProvider client={createDataClient({ fetch: async () => new Response(null, { status: 501 }) })}>
<View slugs={slugs} segments={[]} locale="en" data={JSON.parse(JSON.stringify(data))} />
</DataProvider>,
)
}
const document = (documentId: string, status: string, error: string | null) => ({
documentId,
collectionName: "contracts",
fileId: `file-${documentId}`,
filename: `${documentId}.pdf`,
status,
error,
indexedAt: null,
requestedBy: "u1",
})
const status = (enabled: boolean, state: string, reasons: readonly string[] = [], refusal?: string) => ({
enabled,
service: { state, reasons, ...(refusal === undefined ? {} : { refusal }) },
documents: { queued: 1, indexing: 0, indexed: 2, failed: 0, cancelled: 0 },
})
const answers = {
"knowledge.collection.list": {
collections: [{ name: "contracts", documentCount: 2, indexedCount: 1, createdBy: "u1" }],
},
"knowledge.status": status(true, "degraded", ["the search index is behind"]),
"knowledge.document.list": {
documents: [document("d1", "indexed", null), document("d2", "failed", "the file could not be read")],
},
"knowledge.autoindex.get": { collectionName: null },
}
describe("the knowledge page", () => {
it("is listed for the shell", async () => {
await expect(modulePages.knowledge?.()).resolves.toBe(knowledgePage)
})
it("shows the service's health, the collection and its documents in words", async () => {
const html = await render(answers)
expect(html).toContain("<h1 ")
expect(html).toContain("<li>the search index is behind</li>")
expect(html).toContain(">contracts</h2>")
expect(html).toContain("d1.pdf")
expect(html).toContain("the file could not be read")
expect(html).toContain('role="grid"')
expect(html.match(/role="columnheader"/g)).toHaveLength(3)
})
// core-11: the project's automatic PDF indexing, off or naming its collection, with the project's slugs.
it("offers automatic PDF indexing into one of the collections, showing the current choice", async () => {
const asked: unknown[] = []
const off = await load([], answers, asked)
expect(asked).toContainEqual(["knowledge.autoindex.get", slugs])
expect(off.autoIndex.current).toBeNull()
const html = await render(answers)
expect(html).toContain("Index uploaded PDFs automatically")
const on = await load([], { ...answers, "knowledge.autoindex.get": { collectionName: "contracts" } })
expect(on.autoIndex.current).toBe("contracts")
})
it("does not offer automatic indexing while the module is off", async () => {
const html = await render({ ...answers, "knowledge.status": status(false, "disabled") })
expect(html).not.toContain("Index uploaded PDFs automatically")
})
it("reads each collection's documents with the project's slugs", async () => {
const asked: unknown[] = []
await load([], answers, asked)
expect(asked).toContainEqual(["knowledge.document.list", { ...slugs, collectionName: "contracts" }])
})
it("says the module is off in place of its collections", async () => {
const html = await render({ ...answers, "knowledge.status": status(false, "disabled") })
expect(html).toContain('role="status"')
expect(html).not.toContain('role="grid"')
expect(html).not.toContain("<dl>")
})
it.each([
["plan_required", "The organization's plan does not include document search."],
["feature_disabled", "Document search is switched off for this organization by platform policy."],
["quota_exhausted", "Today's indexing allowance is spent. Try again tomorrow."],
["budget_exhausted", "The organization's AI budget is spent. An administrator can raise it on the platform."],
[
"scope_missing",
"This deployment's token does not hold knowledge.collections:use. Deploy the app again, then have an owner or admin approve it.",
],
["indexing", "The service is building its index. Try again in a few minutes."],
["approval_required", "An administrator must approve this on the platform first."],
["runtime_unavailable", "The knowledge service did not answer. Try again later."],
])("shows the refusal %s as the page's own state", async (refusal, text) => {
const html = await render({
...answers,
"knowledge.status": status(true, "unavailable", [], refusal),
})
expect(html).toContain(`data-state="${refusal}"`)
expect(html).toContain('role="status"')
expect(html).toContain(text.replaceAll("'", "&#x27;"))
})
it("shows a refusal in Lithuanian to a Lithuanian reader", async () => {
const lt = createI18n("lt", {
lt: { knowledge: catalogues.lt?.knowledge ?? {}, app: { failed: "Nepavyko: {detail}" } },
})
const data = await knowledgePage.load({
slugs,
segments: [],
search: {},
caller: callerOf({ ...answers, "knowledge.status": status(true, "disabled", [], "plan_required") }),
i18n: lt,
container: new Container(),
})
expect(data.status.refusal).toEqual({
code: "plan_required",
text: "Organizacijos planas dokumentų paieškos neapima.",
})
})
it("says a document's failure in words, and keeps a code it does not know", async () => {
const html = await render({
...answers,
"knowledge.document.list": {
documents: [document("d1", "failed", "quota_exhausted"), document("d2", "failed", "something_new")],
},
})
// Worded in the browser from the failure labels the loader handed over: the list is live.
expect(html).toContain(">Today&#x27;s indexing allowance is spent. Try again tomorrow.<")
expect(html).toContain(">something_new<")
})
it("answers a path it does not know as not found, without reading", async () => {
const call = vi.fn()
await expect(
knowledgePage.load({
slugs,
segments: ["extra"],
search: {},
caller: { call } as unknown as TransportCaller,
i18n,
container: new Container(),
}),
).rejects.toMatchObject({ status: 404 })
expect(call).not.toHaveBeenCalled()
})
it("hands a procedure's refusal on, as the AppError the shell answers", async () => {
const refused = load([], { ...answers, "knowledge.collection.list": notFound("no such project") })
await expect(refused).rejects.toBeInstanceOf(AppError)
await expect(refused).rejects.toMatchObject({ status: 404 })
})
})
// Explicit synthetic admission: module-local fixtures exercise the selected module.
vi.mock("../../inclusion.json", () => ({
default: { schemaVersion: 1, selected: ["knowledge"], everAdopted: ["knowledge"] },
}))
@@ -0,0 +1,304 @@
// The PDF intake's two adapters (template lesson core-11). The reader hands a PDF to the platform's
// document reader and answers its passages (`@conusai-sdk/ocr/passages`); the passage index sends one
// inline item per passage, under the file's Blob reference plus `#p{page}.{index}` and with its page,
// blocks and file in metadata, and tombstones exactly those references. A search hit on a passage
// cites back into the file it came from.
import { createMemoryDataSource } from "@conusai-kit/data"
import { DEFAULT_FILES_POLICY, files } from "@conusai-kit/files"
import {
KNOWLEDGE_LIMITS,
type Knowledge,
type KnowledgeClient,
KnowledgeError,
type KnowledgeInlineItem,
} from "@conusai-kit/platform"
import { fakeObjectStore } from "@conusai-kit/testing"
import { OcrError } from "@conusai-sdk/ocr"
import { describe, expect, it } from "vitest"
import { platformKnowledgeSearch, platformPassageIndex } from "../adapters/platform-knowledge.js"
import { sdkKnowledgeReader } from "../adapters/reader.js"
import type { KnowledgePassage, KnowledgeScope } from "../core/ports.js"
const scope: KnowledgeScope = {
level: "project",
actor: { userId: "system:knowledge.autoindex", system: true },
orgId: "01900000-0000-7000-0000-000000000001",
workspaceId: "01900000-0000-7000-0000-000000000002",
projectId: "01900000-0000-7000-0000-000000000003",
}
const storageKey = `files/${scope.workspaceId}/${scope.projectId}/01900000-0000-7000-0000-000000000100`
const pathname = `org/${scope.orgId}/${storageKey}`
async function storedFile() {
const data = createMemoryDataSource({ entities: [files] })
const deps = { data, store: fakeObjectStore(), policy: DEFAULT_FILES_POLICY }
const row = await data
.forScope(scope)
.of(files)
.insert({
status: "stored",
storageKey,
filename: "contract.pdf",
contentType: "application/pdf",
size: 10,
sha256: "0".repeat(64),
uploadedBy: "01900000-0000-7000-0000-000000000010",
storedAt: "2026-11-09T19:00:00.000Z",
uploadId: null,
declaredSize: null,
partSize: null,
deletingAt: null,
})
return { deps, fileId: row.id }
}
type PassageClient = Pick<KnowledgeClient, "createCollection" | "indexInline" | "job" | "remove">
function passageClient(jobError: string | null = null) {
const calls = {
collections: [] as string[],
batches: [] as KnowledgeInlineItem[][],
polled: [] as string[],
removed: [] as string[],
}
const client: PassageClient = {
createCollection: async (input) => {
calls.collections.push(`${input.collection}@${input.environment}`)
return input
},
indexInline: async (input) => {
calls.batches.push([...input.items])
return input.items.map((item, at) => ({
documentRef: item.documentRef,
jobId: `job-${calls.batches.length}-${at}`,
}))
},
job: async (jobId) => {
calls.polled.push(jobId)
return jobError === null
? { jobId, status: "succeeded", progress: 1, generation: 1, error: null }
: { jobId, status: "failed", progress: null, generation: null, error: jobError }
},
remove: async (documentRef) => {
calls.removed.push(documentRef)
},
}
return { client, calls }
}
const passage = (page: number, index: number, blockIds: string[] = []): KnowledgePassage => ({
page,
index,
blockIds,
text: `page ${page}, passage ${index}`,
})
describe("the passage index", () => {
it("indexes one tenant item per passage under the file's reference, with its page, blocks and file in metadata", async () => {
const { deps, fileId } = await storedFile()
const { client, calls } = passageClient()
const index = platformPassageIndex(client, deps, { environment: "production", intervalMs: 1 })
const documentRef = await index.reference(scope, fileId)
expect(documentRef).toBe(pathname)
await index.index(scope, {
collectionName: "contracts",
documentRef,
fileId,
passages: [passage(1, 0, ["b1", "b2"]), passage(2, 0)],
})
expect(calls.collections).toEqual(["contracts@production"])
const [batch] = calls.batches
expect(batch?.map((item) => item.documentRef)).toEqual([`${pathname}#p1.0`, `${pathname}#p2.0`])
expect(batch?.[0]).toMatchObject({
text: "page 1, passage 0",
visibility: "tenant",
metadata: { page: "1", block_ids: "b1,b2", file_id: fileId },
})
// A passage on a page with no blocks carries no block ids rather than an empty one.
expect(batch?.[1]?.metadata).toEqual({ page: "2", file_id: fileId })
// The tenant partition is the project's, the same one a search of the project asks for.
expect(new Set(batch?.map((item) => item.tenant))).toEqual(new Set([batch?.[0]?.tenant]))
expect(batch?.[0]?.tenant).toMatch(/^[0-9a-f]{64}$/)
// Every item's durable job is waited for.
expect([...calls.polled].sort()).toEqual(["job-1-0", "job-1-1"])
})
it("sends at most the service's item count per call", async () => {
const { deps, fileId } = await storedFile()
const { client, calls } = passageClient()
const many = Array.from({ length: KNOWLEDGE_LIMITS.inlineItems + 1 }, (_, at) => passage(1, at))
await platformPassageIndex(client, deps, { environment: "production", intervalMs: 1 }).index(scope, {
collectionName: "contracts",
documentRef: pathname,
fileId,
passages: many,
})
expect(calls.batches.map((batch) => batch.length)).toEqual([KNOWLEDGE_LIMITS.inlineItems, 1])
})
it("records a quota refusal as the document's, not as something to retry", async () => {
const { deps, fileId } = await storedFile()
const { client } = passageClient("quota-exhausted")
await expect(
platformPassageIndex(client, deps, { environment: "production", intervalMs: 1 }).index(scope, {
collectionName: "contracts",
documentRef: pathname,
fileId,
passages: [passage(1, 0)],
}),
).rejects.toMatchObject({ reason: "quota_exhausted", permanent: "quota_exhausted" })
})
it("tombstones exactly the references it indexed, without the file", async () => {
const { deps } = await storedFile()
const { client, calls } = passageClient()
await platformPassageIndex(client, deps, { environment: "production" }).remove(scope, {
documentRef: pathname,
passages: [
{ page: 1, index: 0 },
{ page: 1, index: 1 },
{ page: 3, index: 0 },
],
})
expect(calls.removed).toEqual([`${pathname}#p1.0`, `${pathname}#p1.1`, `${pathname}#p3.0`])
})
it("takes a passage the service never indexed as already removed", async () => {
const { deps } = await storedFile()
const { client, calls } = passageClient()
const index = platformPassageIndex(
{
...client,
remove: async (documentRef) => {
calls.removed.push(documentRef)
if (documentRef.endsWith("#p1.1")) throw new KnowledgeError("unavailable_resource", "no such document")
},
},
deps,
{ environment: "production" },
)
await index.remove(scope, {
documentRef: pathname,
passages: [
{ page: 1, index: 0 },
{ page: 1, index: 1 },
{ page: 2, index: 0 },
],
})
expect(calls.removed).toEqual([`${pathname}#p1.0`, `${pathname}#p1.1`, `${pathname}#p2.0`])
})
})
describe("a search hit on a passage", () => {
it("cites the file it came from, with the passage's page and first block", async () => {
const { deps, fileId } = await storedFile()
const client = {
search: async () => [
{
documentRef: `${pathname}#p2.0`,
snippet: "page 2",
score: 0.9,
citations: [],
chunkId: "chunk-1",
metadata: { page: "2", block_ids: "b3,b4", file_id: fileId },
blockIds: [],
},
],
} as unknown as Knowledge
const hits = await platformKnowledgeSearch(client, deps, { environment: "production" }).search(scope, {
collectionName: "contracts",
query: "page",
limit: 5,
actorId: "01900000-0000-7000-0000-000000000010",
})
expect(hits).toEqual([
{
fileId,
filename: "contract.pdf",
collectionName: "contracts",
score: 0.9,
chunk: 1,
snippet: "page 2",
page: 2,
blockId: "b3",
chunkId: "chunk-1",
},
])
})
})
describe("the reader", () => {
const document = {
schema_version: "conusai.ocr.document/v1",
pages: [
{
index: 0,
lines: [
{ text: "Agreement", order: 0, repeated: false },
{ text: "Signed", order: 1, repeated: false },
],
blocks: [{ id: "p1-b1", order: 0, lines: [0, 1] }],
},
],
}
it("submits the PDF once under its key and answers the read document's passages", async () => {
const submitted: Array<{ filename?: string; idempotencyKey?: string }> = []
const reader = sdkKnowledgeReader(
{
submit: (async (_bytes: unknown, options: { filename?: string; idempotencyKey?: string }) => {
submitted.push(options)
return { id: 7 }
}) as never,
wait: (async (job: unknown) => {
expect(job).toBe(7)
return document
}) as never,
},
{ intervalMs: 1 },
)
const read = await reader.read(new Uint8Array([1]), { filename: "contract.pdf", idempotencyKey: "key-1" })
expect(submitted).toEqual([{ filename: "contract.pdf", idempotencyKey: "key-1" }])
expect(read).toEqual([{ page: 1, index: 0, blockIds: ["p1-b1"], text: "Agreement\nSigned" }])
})
it("answers a job the platform failed, or a document it cannot cite, as not indexable", async () => {
const failing = sdkKnowledgeReader({
submit: (async () => ({ id: 1 })) as never,
wait: (async () => {
throw new OcrError("failed", { type: "job-failed", status: 200 })
}) as never,
})
await expect(failing.read(new Uint8Array([1]), { filename: "a.pdf", idempotencyKey: "k" })).rejects.toMatchObject({
permanent: "not_indexable",
})
const longId = {
...document,
pages: [{ ...document.pages[0], blocks: [{ id: "x".repeat(257), order: 0, lines: [0, 1] }] }],
}
const uncitable = sdkKnowledgeReader({
submit: (async () => ({ id: 1 })) as never,
wait: (async () => longId) as never,
})
await expect(uncitable.read(new Uint8Array([1]), { filename: "a.pdf", idempotencyKey: "k" })).rejects.toMatchObject(
{
permanent: "not_indexable",
},
)
})
it("leaves a transient failure to the queue", async () => {
const transient = sdkKnowledgeReader({
submit: (async () => {
throw new OcrError("unavailable", { status: 503 })
}) as never,
wait: (async () => document) as never,
})
const error = await transient.read(new Uint8Array([1]), { filename: "a.pdf", idempotencyKey: "k" }).catch((e) => e)
expect(error).toBeInstanceOf(OcrError)
expect((error as { permanent?: unknown }).permanent).toBeUndefined()
})
})
@@ -0,0 +1,325 @@
import { createMemoryDataSource } from "@conusai-kit/data"
import { DEFAULT_FILES_POLICY, type FilesDeps, files } from "@conusai-kit/files"
import { type Knowledge, KnowledgeError, type KnowledgeSearchHit } from "@conusai-kit/platform"
import { fakeObjectStore } from "@conusai-kit/testing"
import { describe, expect, it } from "vitest"
import { platformKnowledgeSearch } from "../adapters/platform-knowledge.js"
import type { KnowledgeScope } from "../core/ports.js"
const scope: KnowledgeScope = {
level: "project",
actor: { userId: "01900000-0000-7000-0000-000000000009" },
orgId: "01900000-0000-7000-0000-000000000001",
workspaceId: "01900000-0000-7000-0000-000000000002",
projectId: "01900000-0000-7000-0000-000000000003",
}
const otherScope: KnowledgeScope = {
...scope,
projectId: "01900000-0000-7000-0000-000000000004",
}
const storageKey = `files/${scope.workspaceId}/${scope.projectId}/01900000-0000-7000-0000-000000000100`
const otherStorageKey = `files/${otherScope.workspaceId}/${otherScope.projectId}/01900000-0000-7000-0000-000000000101`
const pathname = `org/${scope.orgId}/${storageKey}`
interface ClientStub {
indexed: string[]
removed: string[]
searches: Array<Parameters<Knowledge["search"]>[0]>
results: readonly KnowledgeSearchHit[]
}
function knowledgeClient(stub: ClientStub): Knowledge {
return {
readiness: async () => ({ state: "ready", reasons: [], checkedAt: new Date().toISOString() }),
createCollection: async (input) => input,
search: async (input) => {
stub.searches.push(input)
return stub.results
},
index: async (input) => {
stub.indexed.push(input.documentRef)
return { jobId: "job-1" }
},
job: async (jobId) => ({ jobId, status: "succeeded", progress: 1, generation: 1, error: null }),
cancel: async () => undefined,
remove: async (documentRef) => {
stub.removed.push(documentRef)
},
}
}
function stub(results: readonly KnowledgeSearchHit[] = []): ClientStub {
return { indexed: [], removed: [], searches: [], results }
}
function makeFiles() {
const data = createMemoryDataSource({ entities: [files] })
const deps: FilesDeps = { data, store: fakeObjectStore(), policy: DEFAULT_FILES_POLICY }
return { data, deps }
}
async function seedFile(
data: ReturnType<typeof createMemoryDataSource>,
atScope: KnowledgeScope,
key: string,
filename = "contract.txt",
): Promise<string> {
const row = await data.forScope(atScope).of(files).insert({
status: "stored",
storageKey: key,
filename,
contentType: "text/plain",
size: 17,
sha256: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
uploadedBy: atScope.actor.userId,
storedAt: "2026-10-24T00:00:00.000Z",
uploadId: null,
declaredSize: null,
partSize: null,
deletingAt: null,
})
return row.id
}
describe("the managed knowledge Blob reference", () => {
it("indexes and removes by Blob pathname rather than opaque file id", async () => {
const { data, deps } = makeFiles()
const fileId = await seedFile(data, scope, storageKey)
const client = stub()
const service = platformKnowledgeSearch(knowledgeClient(client), deps, { environment: "preview" })
await service.index(scope, {
collectionName: "contracts",
fileId,
filename: "contract.txt",
bytes: new TextEncoder().encode("managed knowledge"),
})
await service.remove(scope, { collectionName: "contracts", fileId })
expect(client.indexed).toEqual([pathname])
expect(client.removed).toEqual([pathname])
})
it("removes a deleted source by the key it had, whose files row is gone (F407)", async () => {
const { data, deps } = makeFiles()
const fileId = await seedFile(data, scope, storageKey)
await data.forScope(scope).of(files).delete(fileId)
const client = stub()
const service = platformKnowledgeSearch(knowledgeClient(client), deps, { environment: "preview" })
// Without the key the file can no longer be found, so the removal refuses rather than guessing.
await expect(service.remove(scope, { collectionName: "contracts", fileId })).rejects.toThrow()
await service.remove(scope, { collectionName: "contracts", fileId, storageKey })
expect(client.removed).toEqual([pathname])
})
it("maps citations through scoped file rows and drops foreign or stale references", async () => {
const { data, deps } = makeFiles()
const fileId = await seedFile(data, scope, storageKey)
await seedFile(data, otherScope, otherStorageKey, "other-project.txt")
const client = stub([
{ documentRef: pathname, snippet: "own snippet", score: 0.9, citations: [{ page: 1 }] },
{
documentRef: `org/${scope.orgId}/${otherStorageKey}`,
snippet: "other project snippet",
score: 0.8,
citations: [],
},
{
documentRef: `org/01900000-0000-7000-0000-000000000777/${storageKey}`,
snippet: "other organization snippet",
score: 0.7,
citations: [],
},
{
documentRef: `org/${scope.orgId}/files/${scope.workspaceId}/${scope.projectId}/01900000-0000-7000-0000-000000000200`,
snippet: "stale snippet",
score: 0.6,
citations: [],
},
])
const service = platformKnowledgeSearch(knowledgeClient(client), deps, { environment: "preview" })
const hits = await service.search(scope, {
collectionName: "contracts",
query: "contract",
limit: 10,
actorId: scope.actor.userId,
})
expect(hits).toEqual([
{
fileId,
filename: "contract.txt",
collectionName: "contracts",
score: 0.9,
chunk: 1,
snippet: "own snippet",
page: 1,
},
])
expect(client.searches).toHaveLength(1)
expect(client.searches[0]).toMatchObject({ collection: "contracts", query: "contract", limit: 10 })
expect(client.searches[0]?.tenant).toMatch(/^[0-9a-f]{64}$/)
expect(client.searches[0]).not.toHaveProperty("ownerRef")
})
it("refuses a foreign file id before any managed index call", async () => {
const { data, deps } = makeFiles()
const foreignFileId = await seedFile(data, otherScope, otherStorageKey)
const client = stub()
const service = platformKnowledgeSearch(knowledgeClient(client), deps, { environment: "preview" })
await expect(
service.index(scope, {
collectionName: "contracts",
fileId: foreignFileId,
filename: "other-project.txt",
bytes: new TextEncoder().encode("foreign content"),
}),
).rejects.toThrow()
expect(client.indexed).toEqual([])
})
})
describe("the managed service's collections", () => {
it("creates the collection in the deployment's environment before it indexes: the service answers 404 otherwise", async () => {
const { data, deps } = makeFiles()
const fileId = await seedFile(data, scope, storageKey)
const order: string[] = []
const client: Knowledge = {
...knowledgeClient(stub()),
createCollection: async (input) => {
order.push(`collection ${input.collection} in ${input.environment}`)
return input
},
index: async () => {
order.push("index")
return { jobId: "job-1" }
},
}
const input = { collectionName: "contracts", fileId, filename: "a.txt", bytes: new TextEncoder().encode("text") }
await platformKnowledgeSearch(client, deps, { environment: "preview" }).index(scope, input)
expect(order).toEqual(["collection contracts in preview", "index"])
order.length = 0
const missing = platformKnowledgeSearch(client, deps, { environment: undefined })
expect(await missing.readiness(scope)).toMatchObject({ state: "unavailable", refusal: "configuration_missing" })
await expect(missing.index(scope, input)).rejects.toMatchObject({
reason: "configuration_missing",
permanent: "configuration_missing",
})
expect(order).toEqual([])
})
it("answers no results for a collection nothing was indexed into yet", async () => {
const client: Knowledge = {
...knowledgeClient(stub()),
search: async () => {
throw new KnowledgeError("unavailable_resource", "no such collection")
},
}
const hits = await platformKnowledgeSearch(client, makeFiles().deps, { environment: "preview" }).search(scope, {
collectionName: "empty",
query: "anything",
limit: 5,
actorId: scope.actor.userId,
})
expect(hits).toEqual([])
})
})
describe("the managed service's refusals", () => {
const answering = (over: Partial<Knowledge>) => ({ ...knowledgeClient(stub()), ...over })
it.each([
["plan_required", "disabled", "plan_required"],
["feature_disabled", "disabled", "feature_disabled"],
["forbidden", "unavailable", "scope_missing"],
["unauthenticated", "unavailable", "runtime_unavailable"],
["runtime_unavailable", "unavailable", "runtime_unavailable"],
["indexing", "indexing", "indexing"],
] as const)("readiness refused with %s is state %s and the typed refusal %s", async (reason, state, refusal) => {
const client = answering({
readiness: async () => {
throw new KnowledgeError(reason, "the service's own words")
},
})
const answer = await platformKnowledgeSearch(client, makeFiles().deps, { environment: "preview" }).readiness(scope)
expect(answer).toEqual({ state, reasons: [], refusal })
})
it("records a refusal waiting cannot fix on the document, and leaves a transient failure to the queue", async () => {
const { data, deps } = makeFiles()
const fileId = await seedFile(data, scope, storageKey)
const bytes = new TextEncoder().encode("managed knowledge")
const input = { collectionName: "contracts", fileId, filename: "contract.txt", bytes }
const quota = platformKnowledgeSearch(
answering({
job: async (jobId) => ({ jobId, status: "failed", progress: null, generation: null, error: "quota-exhausted" }),
}),
deps,
{ environment: "preview" },
)
await expect(quota.index(scope, input)).rejects.toMatchObject({ permanent: "quota_exhausted" })
const scope_missing = platformKnowledgeSearch(
answering({
index: async () => {
throw new KnowledgeError("forbidden", "the token lacks knowledge.collections:use")
},
}),
deps,
{ environment: "preview" },
)
await expect(scope_missing.index(scope, input)).rejects.toMatchObject({ permanent: "scope_missing" })
const down = platformKnowledgeSearch(
answering({
index: async () => {
throw new KnowledgeError("runtime_unavailable", "try later", true)
},
}),
deps,
{ environment: "preview" },
)
const transient = await down.index(scope, input).catch((error: unknown) => error)
expect(transient).toBeInstanceOf(KnowledgeError)
expect(transient).not.toHaveProperty("permanent")
})
})
describe("bounded durable index polling", () => {
it("stops a queued wait on cancellation without cancelling the durable platform job", async () => {
const { data, deps } = makeFiles()
const fileId = await seedFile(data, scope, storageKey)
const controller = new AbortController()
let polls = 0
let cancellations = 0
const client: Knowledge = {
...knowledgeClient(stub()),
job: async (jobId) => {
polls++
controller.abort()
return { jobId, status: "queued", progress: null, generation: null, error: null }
},
cancel: async () => {
cancellations++
},
}
await expect(
platformKnowledgeSearch(client, deps, { environment: "preview", intervalMs: 1 }).index(scope, {
collectionName: "contracts",
fileId,
filename: "a.txt",
bytes: new Uint8Array(),
signal: controller.signal,
}),
).rejects.toBeDefined()
expect(polls).toBe(1)
expect(cancellations).toBe(0)
})
})
@@ -0,0 +1,144 @@
// Every knowledge procedure a page calls, run as the kernel runs it: `createCaller` hands a handler the
// container of its kind, and that container refuses a port above the kind. `knowledge.status` backs the
// knowledge page and `knowledge.settings.get` the organization's settings page, so a handler that reaches
// for an effect port from a read is a 500 `kind_violation` on those pages on every real stack. The ports
// stay effects: what changes is what each handler asks for.
import { Container, type ProcedureKind, type TenantScope } from "@conusai-kit/core"
import { createMemoryDataSource, DATA_SOURCE, OBJECT_STORE } from "@conusai-kit/data"
import { files } from "@conusai-kit/files"
import { fakeObjectStore } from "@conusai-kit/testing"
import { describe, expect, it, vi } from "vitest"
import {
knowledgeStatusImpl,
settingsDisableDocumentAiImpl,
settingsEnableDocumentAiImpl,
settingsGetImpl,
} from "../adapters/procedures.js"
import { platformKnowledgeReadiness } from "../adapters/readiness.js"
import { KNOWLEDGE_ENTITIES } from "../contract/index.js"
import {
type DocumentAiPlatform,
KNOWLEDGE_ENABLEMENT,
KNOWLEDGE_READINESS,
KNOWLEDGE_SEARCH,
type KnowledgeScope,
} from "../core/ports.js"
const scope: TenantScope & KnowledgeScope = {
level: "project",
actor: { userId: "01900000-0000-7000-0000-000000000009" },
orgId: "01900000-0000-7000-0000-000000000001",
workspaceId: "01900000-0000-7000-0000-000000000002",
projectId: "01900000-0000-7000-0000-000000000003",
}
const projectInput = { org: "acme", workspace: "general", project: "demo" }
const orgInput = { org: "acme" }
const processContainer = () => {
const container = new Container()
container.value(DATA_SOURCE, createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES, files] }))
container.value(OBJECT_STORE, fakeObjectStore())
return container
}
/** A handler as the kernel calls it (`createCaller`): with the container of its procedure's kind. */
const run = (
impl: { readonly kind: ProcedureKind; readonly handler: (call: never) => unknown },
input: unknown,
container: Container,
) => impl.handler({ input, ctx: { container: container.restrict(impl.kind), scope } } as never)
const neverBuilt = () =>
vi.fn((): never => {
throw new Error("a port above this procedure's kind was built")
})
const platform: DocumentAiPlatform = {
checkEligibility: async () => ({ ok: true }),
indexingQuota: async () => 1000,
}
describe("the ports that leave the process", () => {
it("stay refused to a read and a write", () => {
const container = processContainer()
container.value(KNOWLEDGE_ENABLEMENT, platform)
container.register(KNOWLEDGE_SEARCH, neverBuilt())
for (const kind of ["read", "write"] as const) {
expect(() => container.restrict(kind).resolve(KNOWLEDGE_SEARCH)).toThrow(/cannot resolve knowledge.search/)
expect(() => container.restrict(kind).resolve(KNOWLEDGE_ENABLEMENT)).toThrow(
/cannot resolve knowledge.enablement/,
)
}
})
})
describe("the knowledge page's knowledge.status", () => {
it("answers through a read handler's container, and never builds the search service", async () => {
const container = processContainer()
const search = neverBuilt()
container.register(KNOWLEDGE_SEARCH, search)
await expect(run(knowledgeStatusImpl, projectInput, container)).resolves.toMatchObject({
documents: { queued: 0, indexing: 0, indexed: 0, failed: 0, cancelled: 0 },
})
expect(search).not.toHaveBeenCalled()
})
})
describe("the settings page's knowledge.settings", () => {
it("get answers through a read handler's container, without the enablement platform", async () => {
const container = processContainer()
const enablement = neverBuilt()
container.register(KNOWLEDGE_ENABLEMENT, enablement)
await expect(run(settingsGetImpl, orgInput, container)).resolves.toEqual({
enabled: false,
backend: null,
})
expect(enablement).not.toHaveBeenCalled()
})
it("enable reaches the platform and stores the configuration", async () => {
const container = processContainer()
container.value(KNOWLEDGE_ENABLEMENT, platform)
await expect(run(settingsEnableDocumentAiImpl, orgInput, container)).resolves.toEqual({
enabled: true,
backend: "platform",
})
await expect(run(settingsGetImpl, orgInput, container)).resolves.toEqual({ enabled: true, backend: "platform" })
})
it("disable only flips the stored flag, without the enablement platform", async () => {
const container = processContainer()
const enablement = neverBuilt()
container.register(KNOWLEDGE_ENABLEMENT, enablement)
await expect(run(settingsDisableDocumentAiImpl, orgInput, container)).resolves.toEqual({ enabled: false })
expect(enablement).not.toHaveBeenCalled()
})
})
describe("what knowledge.status says of the backend", () => {
it("says what the bound readiness says, and asks it with the request's scope", async () => {
const container = processContainer()
const readiness = vi.fn(async () => ({ state: "degraded" as const, reasons: ["the index is behind"] }))
container.value(KNOWLEDGE_READINESS, { readiness })
await expect(run(knowledgeStatusImpl, projectInput, container)).resolves.toMatchObject({
service: { state: "degraded", reasons: ["the index is behind"] },
})
expect(readiness).toHaveBeenCalledWith(scope)
})
it("says the backend is not bound in a process that has none", async () => {
await expect(run(knowledgeStatusImpl, projectInput, processContainer())).resolves.toMatchObject({
service: { state: "unavailable", reasons: ["the knowledge search service is not bound in this process"] },
})
})
it("reports the managed service failing to answer as unavailable, not as an error page", async () => {
const container = processContainer()
const client = { readiness: vi.fn(async () => Promise.reject(new Error("no answer\nstack"))) }
container.value(KNOWLEDGE_READINESS, platformKnowledgeReadiness(client as never))
await expect(run(knowledgeStatusImpl, projectInput, container)).resolves.toMatchObject({
service: { state: "unavailable", reasons: [], refusal: "runtime_unavailable" },
})
})
})
@@ -0,0 +1,157 @@
import { syntheticNativeReadiness } from "../../../test/support/native-readiness.js"
// A refusal of the platform's knowledge service reaches a person as one typed code: the procedures end to
// end, on the real module, with the service's client refusing underneath; and each code a form looks up.
import { AppError, buildRegistry, Container, createCaller, type TransportCaller } from "@conusai-kit/core"
import { createMemoryDataSource, DATA_SOURCE, entitiesOf, OBJECT_STORE, SYSTEM_DATA } from "@conusai-kit/data"
import { FILES_REGISTRY, filesModule } from "@conusai-kit/files"
import { createI18n } from "@conusai-kit/i18n"
import { KnowledgeError, type KnowledgeReasonCode } from "@conusai-kit/platform"
import { createTenancyHooks, TENANCY, tenancyModule, users } from "@conusai-kit/tenancy"
import { fakeObjectStore } from "@conusai-kit/testing"
import { describe, expect, it } from "vitest"
import { knowledgeRefusal } from "../adapters/refusals.js"
import { KNOWLEDGE_REFUSALS, type KnowledgeRefusal } from "../contract/index.js"
import {
KNOWLEDGE_ENABLEMENT,
KNOWLEDGE_FLAGS,
KNOWLEDGE_READINESS,
KNOWLEDGE_SEARCH,
type KnowledgeSearchService,
} from "../core/ports.js"
import { knowledgeModule } from "../index.js"
import { refusalLabels, refusalOf } from "../ui/labels.js"
async function app(
service: KnowledgeSearchService,
options: { flags?: boolean } = {},
): Promise<{ call: TransportCaller; slugs: Record<string, string> }> {
const registry = buildRegistry([tenancyModule, filesModule, knowledgeModule])
const data = createMemoryDataSource({ entities: entitiesOf(registry, "postgres") })
const root = new Container()
root.value(DATA_SOURCE, data)
root.value(SYSTEM_DATA, data)
root.value(OBJECT_STORE, fakeObjectStore())
root.value(FILES_REGISTRY, registry)
root.value(KNOWLEDGE_SEARCH, service)
root.value(KNOWLEDGE_READINESS, service)
root.value(KNOWLEDGE_ENABLEMENT, { checkEligibility: async () => ({ ok: true }), indexingQuota: async () => 1000 })
if (options.flags !== false) root.value(KNOWLEDGE_FLAGS, { enabled: () => true })
const user = await data.global(users).insert({
issuer: "https://id.example.test",
subject: "owner",
email: "owner@example.test",
displayName: "Owner",
locale: "en",
})
const container = root.child()
const tenancy = createTenancyHooks({ data, registry })
container.value(TENANCY, tenancy)
const call = createCaller(
{ registry, container, tenancy, serviceReadiness: syntheticNativeReadiness },
{ actor: { userId: user.id } },
)
await call.callByName("tenancy.org.create", { slug: "acme", name: "Acme" })
await call.callByName("tenancy.workspace.create", { org: "acme", slug: "ws", name: "WS", visibility: "org" })
await call.callByName("tenancy.project.create", {
org: "acme",
workspace: "ws",
slug: "p",
name: "P",
moduleKey: "knowledge",
})
return { call, slugs: { org: "acme", workspace: "ws", project: "p" } }
}
const refusing = (reason: KnowledgeReasonCode): KnowledgeSearchService => ({
readiness: async () => ({ state: "disabled", reasons: [], refusal: "plan_required" }),
index: async () => undefined,
search: async () => {
throw new KnowledgeError(reason, "the service's own words")
},
remove: async () => undefined,
})
describe("a refused search", () => {
it.each([
["plan_required", "plan_required", 403],
["feature_disabled", "feature_disabled", 403],
["quota_exhausted", "quota_exhausted", 429],
["budget_exhausted", "budget_exhausted", 429],
["forbidden", "scope_missing", 503],
["indexing", "indexing", 503],
["approval_required", "approval_required", 403],
["runtime_unavailable", "runtime_unavailable", 503],
["unauthenticated", "runtime_unavailable", 503],
] as const)("by %s answers %s", async (reason, code, status) => {
const { call, slugs } = await app(refusing(reason))
await call.callByName("knowledge.collection.create", { ...slugs, name: "contracts" })
const refused = await call
.callByName("knowledge.search", { ...slugs, collectionName: "contracts", query: "anything" })
.catch((error: unknown) => error)
expect(refused).toBeInstanceOf(AppError)
expect((refused as AppError).problem).toMatchObject({ code, status })
})
it("leaves a resource the service does not hold as not found, and a programming error as it is", () => {
const missing = knowledgeRefusal(new KnowledgeError("unavailable_resource", "no such collection"))
expect((missing as AppError).problem.status).toBe(404)
const bug = new TypeError("not a refusal")
expect(knowledgeRefusal(bug)).toBe(bug)
})
})
describe("the status a page reads", () => {
it("carries the service's refusal through the read procedure", async () => {
const { call, slugs } = await app(refusing("plan_required"))
expect(await call.callByName("knowledge.status", slugs)).toMatchObject({
enabled: true,
service: { state: "disabled", reasons: [], refusal: "plan_required" },
})
})
})
describe("enabling document AI", () => {
const ready: KnowledgeSearchService = {
readiness: async () => ({ state: "ready", reasons: [] }),
index: async () => undefined,
search: async () => [],
remove: async () => undefined,
}
it("turns knowledge on for the organization's projects, and disabling turns it off again", async () => {
const { call, slugs } = await app(ready, { flags: false })
const enabled = async () => ((await call.callByName("knowledge.status", slugs)) as { enabled: boolean }).enabled
expect(await enabled()).toBe(false)
await call.callByName("knowledge.settings.enableDocumentAi", { org: slugs.org })
expect(await enabled()).toBe(true)
await call.callByName("knowledge.settings.disableDocumentAi", { org: slugs.org })
expect(await enabled()).toBe(false)
})
})
describe("a form's refusal", () => {
const t = (key: string) => key
it("names every typed refusal, and leaves any other error to the form's own text", () => {
const labels = refusalLabels(t)
expect(Object.keys(labels).sort()).toEqual([...KNOWLEDGE_REFUSALS].sort())
const refused = (code: KnowledgeRefusal) => ({ problem: { code } })
expect(refusalOf(labels, refused("scope_missing"))).toEqual({
code: "scope_missing",
text: "refusals.scope_missing",
})
expect(refusalOf(labels, { problem: { code: "not_found" } })).toBeUndefined()
expect(refusalOf(labels, new Error("boom"))).toBeUndefined()
})
it("has a message in both languages, the Lithuanian one its own", () => {
const catalogues = buildRegistry([tenancyModule, filesModule, knowledgeModule]).catalogues
const en = createI18n("en", { en: { knowledge: catalogues.en?.knowledge ?? {} } })
const lt = createI18n("lt", { lt: { knowledge: catalogues.lt?.knowledge ?? {} } })
for (const code of KNOWLEDGE_REFUSALS) {
const english = en.message(`knowledge.refusals.${code}`)
expect(english, code).not.toContain("refusals.")
expect(lt.message(`knowledge.refusals.${code}`), code).not.toBe(english)
}
})
})
@@ -0,0 +1,121 @@
// The enablement flow's semantics: idempotent enable, typed denials from the platform checks, nothing
// provisioned on the app's side, and disable that preserves every row and every indexed document.
import type { TenantScope } from "@conusai-kit/core"
import { createMemoryDataSource } from "@conusai-kit/data"
import { DEFAULT_FILES_POLICY } from "@conusai-kit/files"
import { fakeObjectStore } from "@conusai-kit/testing"
import { describe, expect, it } from "vitest"
import { collections, config, documents, KNOWLEDGE_ENTITIES } from "../contract/index.js"
import { createCollection } from "../core/collections.js"
import type { DocumentAiPlatform } from "../core/ports.js"
import { disableDocumentAi, enableDocumentAi, getSettings } from "../core/settings.js"
const ORG = "01900000-0000-7000-0000-000000000001"
const scope: TenantScope = { level: "org", actor: { userId: "01900000-0000-7000-0000-000000000009" }, orgId: ORG }
const platform = (over: Partial<DocumentAiPlatform> = {}): DocumentAiPlatform => ({
checkEligibility: async () => ({ ok: true }),
indexingQuota: async () => 1000,
...over,
})
const deps = (data: ReturnType<typeof createMemoryDataSource>, p: DocumentAiPlatform) => ({
data,
platform: p,
now: () => Date.UTC(2026, 9, 23),
})
describe("the enablement flow", () => {
it("enables idempotently: the second call writes nothing and answers the same configuration", async () => {
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES] })
const first = await enableDocumentAi(deps(data, platform()), scope)
expect(first).toEqual({ enabled: true, backend: "platform" })
const second = await enableDocumentAi(deps(data, platform()), scope)
expect(second).toEqual(first)
expect((await getSettings(deps(data, platform()), scope)).enabled).toBe(true)
})
it("surfaces the platform's denials as typed reason codes, retirable", async () => {
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES] })
const denied = await enableDocumentAi(
deps(data, platform({ checkEligibility: async () => ({ ok: false, reason: "plan_required" }) })),
scope,
).catch((error: unknown) => error)
expect((denied as { code?: string }).code).toBe("plan_required")
// Nothing was persisted by the denied attempt; the retry path runs the same checks again.
expect((await getSettings(deps(data, platform()), scope)).enabled).toBe(false)
const quota = await enableDocumentAi(deps(data, platform({ indexingQuota: async () => 0 })), scope).catch(
(error: unknown) => error,
)
expect((quota as { code?: string }).code).toBe("quota_exhausted")
})
it("provisions nothing of the app's own: enabling records that the platform serves this organization", async () => {
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES] })
const asked: string[] = []
const watched: DocumentAiPlatform = new Proxy(platform(), {
get: (target, name, receiver) => {
asked.push(String(name))
return Reflect.get(target, name, receiver)
},
})
await enableDocumentAi(deps(data, watched), scope)
expect(asked).toEqual(["checkEligibility", "indexingQuota"])
const rows = await data.forScope(scope).of(config).list({ limit: 1 })
expect(rows.items[0]).toMatchObject({ enabled: true, backend: "platform", serviceId: null, url: null })
expect(await getSettings(deps(data, platform()), scope)).toEqual({ enabled: true, backend: "platform" })
})
it("disable preserves collections, documents and the recorded configuration; re-enable resumes without duplication", async () => {
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES] })
const projectScope: TenantScope = {
level: "project",
actor: { userId: "01900000-0000-7000-0000-000000000009" },
orgId: ORG,
workspaceId: "01900000-0000-7000-0000-000000000002",
projectId: "01900000-0000-7000-0000-000000000003",
}
const knowledgeDepsForProject = {
data,
files: { data, store: fakeObjectStore(), policy: DEFAULT_FILES_POLICY },
flags: { enabled: async () => true },
}
await enableDocumentAi(deps(data, platform()), scope)
const collection = await createCollection(knowledgeDepsForProject, projectScope as never, "contracts")
expect(collection.name).toBe("contracts")
// Seed a document row directly (the upload half is the files module's own suite).
await data.forScope(projectScope).of(documents).insert({
collectionName: "contracts",
fileId: "01900000-0000-7000-0000-000000000099",
filename: "a.txt",
status: "indexed",
error: null,
indexedAt: "2026-10-23T00:00:00.000Z",
requestedBy: "01900000-0000-7000-0000-000000000009",
})
const disabled = await disableDocumentAi(deps(data, platform()), scope)
expect(disabled.enabled).toBe(false)
// Everything is still there.
expect((await getSettings(deps(data, platform()), scope)).backend).toBe("platform")
const collectionsAfter = await data
.forScope(projectScope)
.of(collections)
.list({ where: [{ field: "name", op: "eq", value: "contracts" }], limit: 1 })
expect(collectionsAfter.items).toHaveLength(1)
const documentsAfter = await data
.forScope(projectScope)
.of(documents)
.list({ where: [{ field: "collectionName", op: "eq", value: "contracts" }], limit: 1 })
expect(documentsAfter.items[0]?.status).toBe("indexed")
// Re-enable: idempotent, same row, no second collection.
await enableDocumentAi(deps(data, platform()), scope)
const again = await data
.forScope(projectScope)
.of(collections)
.list({ where: [{ field: "name", op: "eq", value: "contracts" }], limit: 1 })
expect(again.items).toHaveLength(1)
})
})
+86
View File
@@ -0,0 +1,86 @@
// The components' markup, rendered on the server the way a page renders it: what a screen reader
// and a keyboard user get is in the HTML — a name for every control, a status that says what a
// colour would, and the AI disclosure where AI output appears.
import { buildRegistry } from "@conusai-kit/core"
import { builtInCatalogues, createI18n, uiLabels } from "@conusai-kit/i18n"
import { createDataClient, DataProvider } from "@conusai-kit/ui/data"
import { UiLabelsProvider } from "@conusai-kit/ui/primitives"
import { renderToStaticMarkup } from "react-dom/server"
import { describe, expect, it, vi } from "vitest"
import { appModules } from "../../index.js"
import { searchFormLabels } from "../ui/labels.js"
import { SearchForm } from "../ui/search-form.js"
import { SettingsSection } from "../ui/settings-section.js"
const catalogues = buildRegistry([...appModules]).catalogues
const i18n = createI18n("en", { en: { knowledge: catalogues.en?.knowledge ?? {} } })
const t = (key: string, values?: Record<string, string | number>) => i18n.message(`knowledge.${key}`, values)
const failed = "That did not work: {detail}"
const slugs = { org: "acme", workspace: "ws", project: "p" }
const kitStrings = uiLabels(createI18n("en", builtInCatalogues))
const inProvider = (node: React.ReactNode) =>
renderToStaticMarkup(
<UiLabelsProvider labels={kitStrings}>
<DataProvider client={createDataClient({ fetch: async () => new Response(null, { status: 501 }) })}>
{node}
</DataProvider>
</UiLabelsProvider>,
)
describe("the search form", () => {
const html = inProvider(
<SearchForm slugs={slugs} collectionName="contracts" labels={searchFormLabels(t)} locale="en" />,
)
it("names its controls and carries the AI disclosure", () => {
expect(html).toMatch(/<label[^>]*for=/)
expect(html).toContain('type="search"')
expect(html).toContain("kit-ai-disclosure")
expect(html).toContain(t("search.disclosure"))
})
})
describe("the settings section", () => {
const labels = {
heading: t("settings.heading"),
state: t("settings.state"),
stateOn: t("settings.stateOn"),
stateOff: t("settings.stateOff"),
backend: t("settings.backend"),
backendPlatform: t("settings.backendPlatform"),
enable: t("settings.enable"),
enabling: t("settings.enabling"),
disable: t("settings.disable"),
disabling: t("settings.disabling"),
enabledDone: t("settings.enabledDone"),
disabledDone: t("settings.disabledDone"),
planRequired: t("settings.planRequired"),
featureDisabled: t("settings.featureDisabled"),
quotaExhausted: t("settings.quotaExhausted"),
upgrade: t("settings.upgrade"),
hint: t("settings.hint"),
failed,
}
it("renders the state and the enable action when disabled", () => {
const html = inProvider(<SettingsSection org="acme" initial={{ enabled: false, backend: null }} labels={labels} />)
expect(html).toContain(t("settings.stateOff"))
expect(html).toContain(`>${t("settings.enable")}<`)
})
it("says the platform serves the organization, with the disable action, when enabled", () => {
const html = inProvider(
<SettingsSection org="acme" initial={{ enabled: true, backend: "platform" }} labels={labels} />,
)
expect(html).toContain(t("settings.backendPlatform").replaceAll("'", "&#x27;"))
expect(html).toContain(`>${t("settings.disable")}<`)
})
})
// Explicit synthetic admission: module-local fixtures exercise the selected module.
vi.mock("../../inclusion.json", () => ({
default: { schemaVersion: 1, selected: ["knowledge"], everAdopted: ["knowledge"] },
}))
+61
View File
@@ -0,0 +1,61 @@
// The automatic intake's enabling-person metadata is an account-data share. The
// account kernel supplies the subject; this has no public procedure or bearer.
import type { TenantScope } from "@conusai-kit/core"
import { systemActor } from "@conusai-kit/core/worker"
import { DATA_SOURCE, SYSTEM_DATA } from "@conusai-kit/data"
import { type AccountDataContext, type AccountDataContribution, servedProjects } from "@conusai-kit/tenancy"
import { autoIndex } from "../contract/index.js"
async function* ownedOptIns(context: AccountDataContext) {
context.signal?.throwIfAborted()
const data = context.container.resolve(DATA_SOURCE)
const system = context.container.resolve(SYSTEM_DATA)
for await (const project of servedProjects(system)) {
context.signal?.throwIfAborted()
const scope: TenantScope = { level: "project", ...project, actor: systemActor("knowledge.account") }
for (let cursor: string | undefined; ; ) {
const page = await data
.forScope(scope)
.of(autoIndex)
.list({
where: [{ field: "enabledBy", op: "eq", value: context.subject }],
limit: 100,
...(cursor === undefined ? {} : { cursor }),
})
for (const row of page.items) yield { scope, row }
if (!page.hasMore || page.cursor === undefined) break
cursor = page.cursor
}
}
}
export const knowledgeAccountData: AccountDataContribution = {
key: "knowledge",
async export(context) {
const automaticIndexing = []
for await (const { scope, row } of ownedOptIns(context)) {
automaticIndexing.push({
orgId: scope.orgId,
workspaceId: scope.workspaceId,
projectId: scope.projectId,
collectionName: row.collectionName,
enabledAt: row.enabledAt,
})
}
return { automaticIndexing }
},
async erase(context) {
const data = context.container.resolve(DATA_SOURCE)
for await (const { scope, row } of ownedOptIns(context)) {
context.signal?.throwIfAborted()
await data.unitOfWork.run(scope, async (tx) => {
const repository = tx.of(autoIndex)
const current = repository.lock ? await repository.lock(row.id) : await repository.get(row.id)
// A different administrator may have replaced the opt-in since discovery.
if (!current || current.enabledBy !== context.subject) return
if (!repository.lock) await repository.update(current.id, {}, current.version)
await repository.delete(current.id)
})
}
},
}
+85
View File
@@ -0,0 +1,85 @@
// The enablement platform over the platform's own APIs, server-side. Eligibility reads the
// rollout flags (agent_document_tools, agent_knowledge_tools) from the flag snapshot and probes
// the AI tools registry for the document tool family — discovery is the cheap plan/binding/flag
// filter, and "a working OCR endpoint does not by itself make OCR a discoverable tool". Nothing is
// provisioned for the app: the platform's knowledge service owns the index (ADR-061). The quota
// pre-check has no platform API yet (the metering slice owns it); the indexing calls themselves
// carry the typed quota_exhausted refusal at run time.
import { createRegistryClient, ToolIndexingError } from "@conusai-sdk/ai"
import type { EnablementReason } from "../contract/index.js"
import type { DocumentAiPlatform } from "../core/ports.js"
export const AGENT_DOCUMENT_TOOLS_FLAG = "agent_document_tools"
export const AGENT_KNOWLEDGE_TOOLS_FLAG = "agent_knowledge_tools"
const FLAG_KEYS = [AGENT_DOCUMENT_TOOLS_FLAG, AGENT_KNOWLEDGE_TOOLS_FLAG] as const
export interface SdkDocumentAiPlatformOptions {
/** The deployment contract's API URL and token. */
apiUrl: string
token: string
fetch?: typeof fetch
}
export function sdkDocumentAiPlatform(options: SdkDocumentAiPlatformOptions): DocumentAiPlatform {
const http = options.fetch ?? fetch
const rolloutFlagsOff = async (): Promise<string | null> => {
try {
const response = await http(new URL(`${options.apiUrl.replace(/\/+$/, "")}/flags/snapshot`), {
headers: { authorization: `Bearer ${options.token}`, accept: "application/json" },
signal: AbortSignal.timeout(5_000),
})
if (!response.ok) return null // the flags service is not the eligibility authority
const body = (await response.json()) as { flags?: Array<{ key?: string; enabled?: boolean }> }
for (const flag of body.flags ?? []) {
if (
typeof flag.key === "string" &&
(FLAG_KEYS as readonly string[]).includes(flag.key) &&
flag.enabled === false
) {
return flag.key
}
}
return null
} catch {
return null
}
}
return {
async checkEligibility(): Promise<{ ok: true } | { ok: false; reason: EnablementReason; detail?: string }> {
// The rollout flags first: a kill switch or a disabled feature refuses before any probe.
const off = await rolloutFlagsOff()
if (off !== null) return { ok: false, reason: "feature_disabled", detail: `the ${off} flag is off` }
// The plan probe: can this deployment discover the document tool family at all?
try {
const registry = createRegistryClient({
env: { CONUSAI_API_URL: options.apiUrl, CONUSAI_API_TOKEN: options.token },
fetch: http,
})
const found = await registry.tools.search("read and index documents")
if (found.length === 0) {
return { ok: false, reason: "plan_required", detail: "the ai_tools plan capability lists no document tools" }
}
return { ok: true }
} catch (error) {
if (error instanceof ToolIndexingError) {
return { ok: false, reason: "feature_disabled", detail: "the tool registry is still indexing" }
}
return {
ok: false,
reason: "plan_required",
detail: "the AI tools registry did not answer the eligibility probe",
}
}
},
async indexingQuota() {
// No pre-check API exists on the platform yet; the call-time quota refusal is the hard
// gate. Until the limit endpoint lands, the admission check answers "within quota".
return Number.MAX_SAFE_INTEGER
},
}
}
+227
View File
@@ -0,0 +1,227 @@
// The knowledge jobs and the subscribers that enqueue them.
//
// A web request writes the document row and publishes the outbox event in one transaction; the
// module's own subscriber (running in the worker's outbox relay) turns the event into the job,
// with the (collection, file) pair as the singleton key — a redelivered event adds no second job.
//
// The relay delivers every record in its organization's scope, with a system actor, and the worker binds
// `KNOWLEDGE_SCOPE` to that delivery scope. The organization comes from there, never from the payload
// (decision AT-F6); only the workspace and the project come from the record, which this module or the files
// module wrote inside that organization's transaction, as `projectScopeOf` of @conusai-kit/files does. A job,
// unlike a delivery, runs in the project scope it was enqueued in.
import { type EventContext, internal, type JobDef, serviceRefusal, subscribe } from "@conusai-kit/core"
import { DATA_SOURCE, JOB_QUEUE } from "@conusai-kit/data"
import { fileDeleted, type ProcessorJobPayload, processorJobPayloadSchema } from "@conusai-kit/files"
import { NATIVE_JOB_ADMISSION } from "@conusai-kit/platform"
import { projects, workspaces } from "@conusai-kit/tenancy"
import { z } from "zod"
import {
AUTO_INDEX_JOB,
documentIndexRequested,
documentRemoveRequested,
INDEX_JOB,
REMOVE_JOB,
} from "../contract/index.js"
import { autoIndexUpload } from "../core/auto-index-job.js"
import { requestSourceRemoval } from "../core/documents.js"
import { indexDocument, removeDocument } from "../core/index-job.js"
import { KNOWLEDGE_SCOPE, type KnowledgeScope } from "../core/ports.js"
import { knowledgeDeps, knowledgeSearchDeps } from "./procedures.js"
export const documentPayload = z.object({
documentId: z.string(),
collectionName: z.string(),
fileId: z.string(),
/** A remove job's: the object to remove, resolved when the removal was asked for; absent only on an older record. */
storageKey: z.string().optional(),
})
export type DocumentJobPayload = z.output<typeof documentPayload>
export const indexJobKey = (collectionName: string, fileId: string): string =>
`knowledge.index.${collectionName}.${fileId}`
export const removeJobKey = (collectionName: string, fileId: string): string =>
`knowledge.remove.${collectionName}.${fileId}`
/**
* The project a delivery acts in: the delivery's organization and actor, and the record's workspace and project.
* A record that names another organization is refused here rather than acted on there, and so is one whose workspace
* the organization does not have or whose project is not in that workspace: both are read in the delivery's own
* organization before anything is enqueued or changed. That refusal can never succeed on a retry; the relay has no
* terminal-failure path yet, so it is retried until marked stuck, with the reason (a kit-jobs gap, recorded in the
* template lessons). An archived project is acted on: an archive hides a project and does not end its work
* (kit-tenancy `suspension.ts`).
*/
async function projectScopeOf(
ctx: EventContext,
payload: { orgId: string; workspaceId: string; projectId: string },
): Promise<KnowledgeScope & { level: "project" }> {
const ambient = ctx.container.resolve(KNOWLEDGE_SCOPE)
if (ambient.orgId !== payload.orgId) {
throw internal(`a knowledge event of organization ${payload.orgId} was delivered in ${ambient.orgId}`)
}
const data = ctx.container.resolve(DATA_SOURCE)
const inOrg = { level: "org" as const, actor: ambient.actor, orgId: ambient.orgId }
if (!(await data.forScope(inOrg).of(workspaces).get(payload.workspaceId))) {
throw internal(
`a knowledge event names workspace ${payload.workspaceId}, which organization ${ambient.orgId} does not have`,
)
}
const inWorkspace = {
level: "workspace" as const,
actor: ambient.actor,
orgId: ambient.orgId,
workspaceId: payload.workspaceId,
}
if (!(await data.forScope(inWorkspace).of(projects).get(payload.projectId))) {
throw internal(
`a knowledge event names project ${payload.projectId}, which is not in workspace ${payload.workspaceId}`,
)
}
return {
level: "project",
actor: ambient.actor,
orgId: ambient.orgId,
workspaceId: payload.workspaceId,
projectId: payload.projectId,
}
}
export const indexJob: JobDef<DocumentJobPayload> = {
payload: documentPayload,
async handler(payload, ctx) {
const scope = ctx.container.resolve(KNOWLEDGE_SCOPE)
if (scope.level !== "project") throw internal(`a knowledge job runs in a project scope, not a ${scope.level} one`)
await indexDocument(
{
...knowledgeSearchDeps(ctx.container),
beforeIndex: async (subject) => {
if (!ctx.signal || !ctx.container.has(NATIVE_JOB_ADMISSION)) throw serviceRefusal("observation_missing")
ctx.signal.throwIfAborted()
await ctx.container.resolve(NATIVE_JOB_ADMISSION).authorize(scope, INDEX_JOB, subject, ctx.signal)
},
},
scope,
payload.documentId,
{
...(ctx.signal === undefined ? {} : { signal: ctx.signal }),
...(ctx.finalAttempt === undefined ? {} : { finalAttempt: ctx.finalAttempt }),
},
)
},
}
/**
* The automatic PDF intake (core-11): the files module enqueues it for every stored upload this module's
* processor wants, in the upload's project. It acts for the administrator who opted the project in, so
* the worker's native admission checks that administrator's permission and the platform's services again
* before each step that leaves the process, as the index job does for the person who asked.
*/
export const autoIndexJob: JobDef<ProcessorJobPayload> = {
payload: processorJobPayloadSchema,
async handler(payload, ctx) {
const scope = ctx.container.resolve(KNOWLEDGE_SCOPE)
if (scope.level !== "project") throw internal(`a knowledge job runs in a project scope, not a ${scope.level} one`)
await autoIndexUpload(
{
...knowledgeSearchDeps(ctx.container),
beforeIndex: async (subject) => {
if (!ctx.signal || !ctx.container.has(NATIVE_JOB_ADMISSION)) throw serviceRefusal("observation_missing")
ctx.signal.throwIfAborted()
await ctx.container.resolve(NATIVE_JOB_ADMISSION).authorize(scope, AUTO_INDEX_JOB, subject, ctx.signal)
},
},
scope,
payload,
{
...(ctx.signal === undefined ? {} : { signal: ctx.signal }),
...(ctx.finalAttempt === undefined ? {} : { finalAttempt: ctx.finalAttempt }),
},
)
},
}
export const removeJob: JobDef<DocumentJobPayload> = {
payload: documentPayload,
async handler(payload, ctx) {
const scope = ctx.container.resolve(KNOWLEDGE_SCOPE)
if (scope.level !== "project") throw internal(`a knowledge job runs in a project scope, not a ${scope.level} one`)
await removeDocument(knowledgeSearchDeps(ctx.container), scope, payload.documentId, payload.storageKey)
},
}
async function onDocumentIndexRequested(
event: {
name: string
payload: {
orgId: string
workspaceId: string
projectId: string
documentId: string
collectionName: string
fileId: string
}
},
ctx: EventContext,
): Promise<void> {
const scope = await projectScopeOf(ctx, event.payload)
await ctx.container.resolve(JOB_QUEUE).enqueue(
scope,
INDEX_JOB,
{
documentId: event.payload.documentId,
collectionName: event.payload.collectionName,
fileId: event.payload.fileId,
},
{
key: indexJobKey(event.payload.collectionName, event.payload.fileId),
},
)
}
async function onDocumentRemoveRequested(
event: {
name: string
payload: {
orgId: string
workspaceId: string
projectId: string
documentId: string
collectionName: string
fileId: string
storageKey?: string | undefined
}
},
ctx: EventContext,
): Promise<void> {
const scope = await projectScopeOf(ctx, event.payload)
await ctx.container.resolve(JOB_QUEUE).enqueue(
scope,
REMOVE_JOB,
{
documentId: event.payload.documentId,
collectionName: event.payload.collectionName,
fileId: event.payload.fileId,
...(event.payload.storageKey === undefined ? {} : { storageKey: event.payload.storageKey }),
},
{
key: removeJobKey(event.payload.collectionName, event.payload.fileId),
},
)
}
async function onSourceDeleted(
event: { payload: { orgId: string; workspaceId: string; projectId: string; fileId: string; storageKey: string } },
ctx: EventContext,
): Promise<void> {
const scope = await projectScopeOf(ctx, event.payload)
// The file's row is gone by now: the key it had travels with the removal (F407).
await requestSourceRemoval(knowledgeDeps(ctx.container), scope, event.payload.fileId, event.payload.storageKey)
}
export const KNOWLEDGE_JOBS = { [INDEX_JOB]: indexJob, [REMOVE_JOB]: removeJob, [AUTO_INDEX_JOB]: autoIndexJob }
export const KNOWLEDGE_SUBSCRIPTIONS = [
subscribe(fileDeleted, onSourceDeleted),
subscribe(documentIndexRequested, onDocumentIndexRequested),
subscribe(documentRemoveRequested, onDocumentRemoveRequested),
]
@@ -0,0 +1,280 @@
// The managed search service: the platform's documents-index service through the kit-platform
// Knowledge client. Indexing and search run server-side there (the service owns chunking,
// embeddings and the vector store); this module supplies the job lifecycle and the citations it
// answers with.
//
// Contract notes (the service's real shapes): the index submission carries the content's SHA-256
// (computed here from the job's bytes) as its version assertion; a quota denial surfaces on the
// job, which the poll below reads as the job's typed error. Blob pathnames and returned citations
// resolve through the files module's scoped rows, so an opaque file id never becomes a pathname
// and a foreign or stale service reference never becomes a valid app hit.
import { createHash } from "node:crypto"
import { setTimeout as delay } from "node:timers/promises"
import {
type FilesDeps,
findFileStorageByKey,
getFileStorage,
platformBlobPathname,
platformBlobStorageKey,
} from "@conusai-kit/files"
import { KNOWLEDGE_LIMITS, type Knowledge, type KnowledgeClient, KnowledgeError } from "@conusai-kit/platform"
import type { KnowledgeHit, KnowledgePassageIndex, KnowledgeScope, KnowledgeSearchService } from "../core/ports.js"
import { platformKnowledgeReadiness } from "./readiness.js"
import { permanentRefusal } from "./refusals.js"
/** The service's tenant partition of a project: a search of the project and its indexed documents agree on it. */
const tenantOf = (scope: KnowledgeScope): string =>
createHash("sha256")
.update(JSON.stringify([scope.orgId, scope.workspaceId, scope.projectId]))
.digest("hex")
/** A passage's reference: the file's own plus `#p{page}.{index}` (the automatic PDF intake, core-11). */
const passageRef = (documentRef: string, passage: { page: number; index: number }): string =>
`${documentRef}#p${passage.page}.${passage.index}`
const PASSAGE_REF = /^(.+)#p([1-9][0-9]*)\.(0|[1-9][0-9]*)$/
/**
* `environment` is the deployment's (`CONUSAI_ENVIRONMENT`): the service keeps a collection in one environment,
* and a deployment token reaches only its own.
*/
export function platformKnowledgeSearch(
client: Knowledge,
files: FilesDeps,
options: { environment?: string | undefined; timeoutMs?: number; intervalMs?: number } = {},
): KnowledgeSearchService {
const environment = options.environment?.trim()
const configuredEnvironment = (): string => {
if (!environment)
throw new KnowledgeError("configuration_missing", "CONUSAI_ENVIRONMENT is required for knowledge indexing")
return environment
}
const unwrap = (error: unknown): never => {
// The client's errors are already typed by reason; a non-knowledge failure is unavailable.
if (error instanceof KnowledgeError) throw error
throw new KnowledgeError(
"runtime_unavailable",
error instanceof Error ? error.message : "the knowledge service failed",
true,
)
}
/** The same, for an index job: a refusal waiting cannot fix is recorded on the document, not retried. */
const unwrapJob = (error: unknown): never => {
if (error instanceof KnowledgeError) {
const permanent = permanentRefusal(error)
if (permanent !== undefined) throw Object.assign(error, { permanent })
}
return unwrap(error)
}
const documentRef = async (scope: KnowledgeScope, fileId: string): Promise<string> => {
const reference = await getFileStorage(files, scope, fileId)
return platformBlobPathname(scope, reference.storageKey)
}
return {
readiness(scope) {
if (!environment)
return Promise.resolve({
state: "unavailable",
reasons: ["CONUSAI_ENVIRONMENT is not set"],
refusal: "configuration_missing",
})
return platformKnowledgeReadiness(client).readiness(scope)
},
async index(scope, input) {
try {
// The service answers 404 for a collection it does not hold. An app's collection is only a row of
// this module (a write cannot leave the process), so the first index creates it; creating it again
// finds it.
const activeEnvironment = configuredEnvironment()
const reference = await documentRef(scope, input.fileId)
const signal = AbortSignal.any([
AbortSignal.timeout(options.timeoutMs ?? 300_000),
...(input.signal ? [input.signal] : []),
])
signal.throwIfAborted()
await client.createCollection({ collection: input.collectionName, environment: activeEnvironment }, { signal })
const { jobId } = await client.index(
{
collection: input.collectionName,
documentRef: reference,
options: {
visibility: "tenant",
tenant: tenantOf(scope),
contentSha256: createHash("sha256").update(input.bytes).digest("hex"),
},
},
{ signal },
)
// The service's job is durable; poll until it leaves the queue. A caller-side cancel is
// the job's own AbortSignal, checked between polls.
for (;;) {
signal.throwIfAborted()
const job = await client.job(jobId, { signal })
if (job.status === "succeeded") return
if (job.status === "failed")
throw new KnowledgeError(
job.error === "quota-exhausted" ? "quota_exhausted" : "runtime_unavailable",
job.error ?? "the indexing job failed",
)
if (job.status === "cancelled")
throw new KnowledgeError("runtime_unavailable", "the indexing job was cancelled")
await delay(options.intervalMs ?? 1_000, undefined, { signal })
}
} catch (error) {
return unwrapJob(error)
}
},
async search(scope, input) {
try {
configuredEnvironment()
const results = await client.search({
collection: input.collectionName,
query: input.query,
limit: input.limit,
tenant: tenantOf(scope),
})
const hits: KnowledgeHit[] = []
for (const [at, hit] of results.entries()) {
// A passage of a PDF the intake indexed names its file's reference, and cites from its metadata:
// the service reports pages only for sources it parsed itself.
const passage = PASSAGE_REF.exec(hit.documentRef)
const storageKey = platformBlobStorageKey(scope, passage?.[1] ?? hit.documentRef)
if (storageKey === null) continue
const reference = await findFileStorageByKey(files, scope, storageKey)
if (reference === null) continue
const metadata = "metadata" in hit ? (hit.metadata as Readonly<Record<string, string>>) : {}
if (passage && metadata.file_id !== reference.file.fileId) continue
const page = passage ? Number(passage[2]) : hit.citations[0]?.page
const blockId = passage ? metadata.block_ids?.split(",")[0] || undefined : hit.citations[0]?.blockId
hits.push({
fileId: reference.file.fileId,
filename: reference.file.filename,
collectionName: input.collectionName,
score: hit.score,
chunk: at + 1,
snippet: hit.snippet,
...(page === undefined ? {} : { page }),
...(blockId === undefined ? {} : { blockId }),
...("chunkId" in hit && typeof hit.chunkId === "string" ? { chunkId: hit.chunkId } : {}),
})
}
return hits
} catch (error) {
// Nothing has been indexed into this collection yet, so the service has not heard of it.
if (error instanceof KnowledgeError && error.reason === "unavailable_resource") return []
return unwrap(error)
}
},
async remove(scope, input) {
try {
// Every removal asked for since F407 names its object; only a record from before it is looked up.
const reference =
input.storageKey === undefined
? await documentRef(scope, input.fileId)
: platformBlobPathname(scope, input.storageKey)
await client.remove(reference)
} catch (error) {
return unwrap(error)
}
},
}
}
/**
* The automatic PDF intake's index (core-11): one inline item per passage, in the project's tenant partition,
* under the file's Blob reference plus `#p{page}.{index}`, with its page, its blocks and its file in metadata.
* Items go at most the service's count per call, and every item's durable job is waited for; a refusal waiting
* cannot fix is the document's, as with a whole file.
*/
export function platformPassageIndex(
client: Pick<KnowledgeClient, "createCollection" | "indexInline" | "job" | "remove">,
files: FilesDeps,
options: { environment?: string | undefined; timeoutMs?: number; intervalMs?: number } = {},
): KnowledgePassageIndex {
const environment = options.environment?.trim()
const failure = (error: unknown): never => {
if (error instanceof KnowledgeError) {
const permanent = permanentRefusal(error)
if (permanent !== undefined) throw Object.assign(error, { permanent })
throw error
}
throw new KnowledgeError(
"runtime_unavailable",
error instanceof Error ? error.message : "the knowledge service failed",
true,
)
}
return {
async reference(scope, fileId) {
const stored = await getFileStorage(files, scope, fileId)
return platformBlobPathname(scope, stored.storageKey)
},
async index(scope, input) {
try {
if (!environment)
throw new KnowledgeError("configuration_missing", "CONUSAI_ENVIRONMENT is required for knowledge indexing")
const signal = AbortSignal.any([
AbortSignal.timeout(options.timeoutMs ?? 300_000),
...(input.signal ? [input.signal] : []),
])
await client.createCollection({ collection: input.collectionName, environment }, { signal })
const tenant = tenantOf(scope)
const jobs: string[] = []
for (let at = 0; at < input.passages.length; at += KNOWLEDGE_LIMITS.inlineItems) {
signal.throwIfAborted()
const accepted = await client.indexInline({
collection: input.collectionName,
items: input.passages.slice(at, at + KNOWLEDGE_LIMITS.inlineItems).map((passage) => ({
documentRef: passageRef(input.documentRef, passage),
text: passage.text,
visibility: "tenant" as const,
tenant,
metadata: {
page: String(passage.page),
...(passage.blockIds.length === 0 ? {} : { block_ids: passage.blockIds.join(",") }),
file_id: input.fileId,
},
})),
})
jobs.push(...accepted.map((job) => job.jobId))
}
for (const jobId of jobs) {
for (;;) {
signal.throwIfAborted()
const job = await client.job(jobId, { signal })
if (job.status === "succeeded") break
if (job.status === "failed")
throw new KnowledgeError(
job.error === "quota-exhausted" ? "quota_exhausted" : "runtime_unavailable",
job.error ?? "the indexing job failed",
)
if (job.status === "cancelled")
throw new KnowledgeError("runtime_unavailable", "the indexing job was cancelled")
await delay(options.intervalMs ?? 1_000, undefined, { signal })
}
}
} catch (error) {
return failure(error)
}
},
async remove(_scope, input) {
for (const passage of input.passages) {
try {
await client.remove(passageRef(input.documentRef, passage))
} catch (error) {
// A passage the service never indexed (the run was refused before it) is already removed.
if (error instanceof KnowledgeError && error.reason === "unavailable_resource") continue
failure(error)
}
}
},
}
}
+154
View File
@@ -0,0 +1,154 @@
// The handlers of the knowledge module's procedures: each resolves its ports from the request's
// container and calls one use case. No rule lives here.
import type { Container } from "@conusai-kit/core"
import { implement } from "@conusai-kit/core"
import { DATA_SOURCE } from "@conusai-kit/data"
import { filesDeps } from "@conusai-kit/files"
import {
autoIndexGet,
autoIndexSet,
collectionCreate,
collectionList,
documentGetMany,
documentIndex,
documentList,
documentRemove,
knowledgeSearch,
knowledgeStatus,
settingsDisableDocumentAi,
settingsEnableDocumentAi,
settingsGet,
} from "../contract/index.js"
import { getAutoIndex, setAutoIndex } from "../core/auto-index.js"
import { createCollection, listCollections } from "../core/collections.js"
import { getManyDocuments, listDocuments, requestIndex, requestRemove } from "../core/documents.js"
import {
flagsOf,
KNOWLEDGE_ENABLEMENT,
KNOWLEDGE_PASSAGES,
KNOWLEDGE_READER,
KNOWLEDGE_READINESS,
KNOWLEDGE_SEARCH,
type KnowledgeDeps,
} from "../core/ports.js"
import { searchKnowledge } from "../core/search.js"
import { disableDocumentAi, enableDocumentAi, getSettings, type SettingsDeps } from "../core/settings.js"
import { type KnowledgeStatusDeps, knowledgeStatusOf } from "../core/status.js"
import { knowledgeRefusal } from "./refusals.js"
/** The ports a write or read handler needs. Never the search service: that port is an effect, and
* resolving it from either is a kind violation. */
export function knowledgeDeps(container: Container): KnowledgeDeps {
const data = container.resolve(DATA_SOURCE)
return { data, files: filesDeps(container), flags: flagsOf(container, data) }
}
/**
* The effect handler's deps: the search service, bound in the processes that may leave the process, and the
* PDF intake's reader, passage index and enablement checks where the worker binds the reader (the jobs; a
* request never reads a PDF).
*/
export function knowledgeSearchDeps(container: Container): KnowledgeDeps {
return {
...knowledgeDeps(container),
search: container.resolve(KNOWLEDGE_SEARCH),
...(container.has(KNOWLEDGE_READER) ? { reader: container.resolve(KNOWLEDGE_READER) } : {}),
...(container.has(KNOWLEDGE_PASSAGES) ? { passages: container.resolve(KNOWLEDGE_PASSAGES) } : {}),
...(container.has(KNOWLEDGE_READER) ? { enablement: container.resolve(KNOWLEDGE_ENABLEMENT) } : {}),
}
}
/** The status read's deps: the backend's readiness, which a read may resolve, instead of the search service. */
function knowledgeStatusDeps(container: Container): KnowledgeStatusDeps {
const data = container.resolve(DATA_SOURCE)
return {
data,
flags: flagsOf(container, data),
...(container.has(KNOWLEDGE_READINESS) ? { readiness: container.resolve(KNOWLEDGE_READINESS) } : {}),
}
}
/** The enablement flow's deps, for the effect that leaves the process. The other settings procedures read and write the module's own row. */
function settingsDeps(container: Container): SettingsDeps {
return { data: container.resolve(DATA_SOURCE), platform: container.resolve(KNOWLEDGE_ENABLEMENT) }
}
const settingsDataOf = (container: Container) => ({ data: container.resolve(DATA_SOURCE) })
export const collectionListImpl = implement(collectionList, ({ ctx }) =>
listCollections(knowledgeDeps(ctx.container), ctx.scope),
)
export const collectionCreateImpl = implement(collectionCreate, ({ input, ctx }) =>
createCollection(knowledgeDeps(ctx.container), ctx.scope, input.name),
)
export const documentIndexImpl = implement(documentIndex, ({ input, ctx }) =>
requestIndex(knowledgeDeps(ctx.container), ctx.scope, input),
)
export const documentListImpl = implement(documentList, ({ input, ctx }) =>
listDocuments(knowledgeDeps(ctx.container), ctx.scope, input.collectionName),
)
export const documentGetManyImpl = implement(documentGetMany, async ({ input, ctx }) => ({
items: await getManyDocuments(knowledgeDeps(ctx.container), ctx.scope, input.ids),
}))
export const documentRemoveImpl = implement(documentRemove, ({ input, ctx }) =>
requestRemove(knowledgeDeps(ctx.container), ctx.scope, input),
)
export const knowledgeSearchImpl = implement(knowledgeSearch, async ({ input, ctx }) => {
try {
return await searchKnowledge(knowledgeSearchDeps(ctx.container), ctx.scope, {
collectionName: input.collectionName,
query: input.query,
...(input.limit === undefined ? {} : { limit: input.limit }),
})
} catch (error) {
throw knowledgeRefusal(error)
}
})
export const knowledgeStatusImpl = implement(knowledgeStatus, ({ ctx }) =>
knowledgeStatusOf(knowledgeStatusDeps(ctx.container), ctx.scope),
)
export const settingsGetImpl = implement(settingsGet, ({ ctx }) =>
getSettings(settingsDataOf(ctx.container), ctx.scope),
)
export const settingsEnableDocumentAiImpl = implement(settingsEnableDocumentAi, async ({ ctx }) => {
const settings = await enableDocumentAi(settingsDeps(ctx.container), ctx.scope)
return { enabled: settings.enabled, backend: settings.backend ?? "platform" }
})
export const settingsDisableDocumentAiImpl = implement(settingsDisableDocumentAi, ({ ctx }) =>
disableDocumentAi(settingsDataOf(ctx.container), ctx.scope),
)
export const autoIndexGetImpl = implement(autoIndexGet, ({ ctx }) =>
getAutoIndex(settingsDataOf(ctx.container), ctx.scope),
)
export const autoIndexSetImpl = implement(autoIndexSet, ({ input, ctx }) =>
setAutoIndex(knowledgeDeps(ctx.container), ctx.scope, input.collectionName),
)
export const KNOWLEDGE_PROCEDURE_IMPLS = [
collectionListImpl,
collectionCreateImpl,
documentIndexImpl,
documentListImpl,
documentGetManyImpl,
documentRemoveImpl,
knowledgeSearchImpl,
knowledgeStatusImpl,
settingsGetImpl,
settingsEnableDocumentAiImpl,
settingsDisableDocumentAiImpl,
autoIndexGetImpl,
autoIndexSetImpl,
]
+46
View File
@@ -0,0 +1,46 @@
// The automatic PDF intake's reader (core-11): the platform's document reader (`DocumentReader`, kit-platform's
// `@conusai-sdk/ocr` wired to the deployment contract) and the read document's passages
// (`@conusai-sdk/ocr/passages`), cut to the knowledge service's limit for one indexed item. The one thing this
// adapter decides is what is permanent: a job the platform failed, or a document whose blocks no metadata
// value can cite (`passages` throws `RangeError`), is not indexable; anything else is thrown for the queue.
import type { DocumentReader } from "@conusai-kit/platform"
import { KNOWLEDGE_LIMITS } from "@conusai-kit/platform"
import { OcrError } from "@conusai-sdk/ocr"
import { passages } from "@conusai-sdk/ocr/passages"
import type { KnowledgeReader } from "../core/ports.js"
export interface SdkKnowledgeReaderOptions {
/** Poll interval of `wait`, in milliseconds. */
readonly intervalMs?: number
/** Give up on one job after this long; the queue's retry submits again under the same key. */
readonly timeoutMs?: number
}
export function sdkKnowledgeReader(
reader: Pick<DocumentReader, "submit" | "wait">,
options: SdkKnowledgeReaderOptions = {},
): KnowledgeReader {
return {
async read(bytes, { filename, idempotencyKey, signal }) {
try {
const job = await reader.submit(bytes, { filename, idempotencyKey })
const document = await reader.wait(job.id, {
...(options.intervalMs === undefined ? {} : { intervalMs: options.intervalMs }),
...(options.timeoutMs === undefined ? {} : { timeoutMs: options.timeoutMs }),
...(signal ? { signal } : {}),
})
return passages(document, { maxBytes: KNOWLEDGE_LIMITS.inlineTextBytes }).map((passage) => ({
page: passage.page,
index: passage.index,
blockIds: passage.blockIds,
text: passage.text,
}))
} catch (error) {
if ((error instanceof OcrError && error.type === "job-failed") || error instanceof RangeError)
throw Object.assign(error, { permanent: "not_indexable" })
throw error
}
},
}
}
+25
View File
@@ -0,0 +1,25 @@
// The platform service's readiness, on its own so the status read can ask it without the search service
// (`KNOWLEDGE_READINESS`): the search adapter answers the same way, from here.
import { type Knowledge, KnowledgeError } from "@conusai-kit/platform"
import type { KnowledgeReadiness } from "../core/ports.js"
import { refusalOfReason, stateOfRefusal } from "./refusals.js"
/**
* The platform's managed service answers for itself. A refusal is the service's answer, as a typed code
* the page shows in the reader's language; a failure to ask it is its being unavailable.
*/
export const platformKnowledgeReadiness = (client: Pick<Knowledge, "readiness">): KnowledgeReadiness => ({
async readiness() {
try {
const answer = await client.readiness()
return { state: answer.state, reasons: [...answer.reasons] }
} catch (error) {
if (error instanceof KnowledgeError) {
const refusal = refusalOfReason(error.reason)
return { state: stateOfRefusal(refusal), reasons: [], ...(refusal === undefined ? {} : { refusal }) }
}
return { state: "unavailable", reasons: [], refusal: "runtime_unavailable" }
}
},
})
+77
View File
@@ -0,0 +1,77 @@
// A refusal of the platform's knowledge service as this module's one closed set of codes
// (`KNOWLEDGE_REFUSALS`), which the pages and forms show in the reader's language. A refusal that is the
// deployment's or the platform's says so; the service's own words stay in `detail`, for the log and the API.
import { AppError, notFound } from "@conusai-kit/core"
import { KnowledgeError, type KnowledgeReasonCode } from "@conusai-kit/platform"
import type { KnowledgeRefusal, ServiceState } from "../contract/index.js"
const REFUSAL: Readonly<Record<Exclude<KnowledgeReasonCode, "unavailable_resource">, KnowledgeRefusal>> = {
plan_required: "plan_required",
feature_disabled: "feature_disabled",
quota_exhausted: "quota_exhausted",
budget_exhausted: "budget_exhausted",
// The service refused the token itself: `knowledge.collections:use` is missing, `*` never carries it.
forbidden: "scope_missing",
indexing: "indexing",
approval_required: "approval_required",
runtime_unavailable: "runtime_unavailable",
configuration_missing: "configuration_missing",
// The platform refused the token's identity: nothing the person can act on, the app is not served.
unauthenticated: "runtime_unavailable",
}
const STATUS: Readonly<Record<KnowledgeRefusal, number>> = {
plan_required: 403,
feature_disabled: 403,
approval_required: 403,
quota_exhausted: 429,
budget_exhausted: 429,
scope_missing: 503,
indexing: 503,
runtime_unavailable: 503,
configuration_missing: 503,
}
const TITLE: Readonly<Record<KnowledgeRefusal, string>> = {
plan_required: "The organization's plan does not include document search",
feature_disabled: "Document search is disabled by policy",
approval_required: "The platform asks for an approval first",
quota_exhausted: "The indexing quota is exhausted",
budget_exhausted: "The AI budget is exhausted",
scope_missing: "This deployment's token does not hold knowledge.collections:use",
indexing: "The knowledge service is building its index",
runtime_unavailable: "The knowledge service is unavailable",
configuration_missing: "The knowledge deployment configuration is incomplete",
}
/** The typed refusal of a service error; a resource it does not hold has none, it is simply not there. */
export function refusalOfReason(reason: KnowledgeReasonCode): KnowledgeRefusal | undefined {
return reason === "unavailable_resource" ? undefined : REFUSAL[reason]
}
/** The readiness state a refusal puts the service in: a plan or a policy is "disabled", not "down". */
export function stateOfRefusal(refusal: KnowledgeRefusal | undefined): ServiceState {
if (refusal === "plan_required" || refusal === "feature_disabled") return "disabled"
return refusal === "indexing" ? "indexing" : "unavailable"
}
/** The app error for anything the service threw; a programming error passes through as it is. */
export function knowledgeRefusal(error: unknown): unknown {
if (!(error instanceof KnowledgeError)) return error
const refusal = refusalOfReason(error.reason)
if (refusal === undefined) return notFound(error.message)
return new AppError({
type: `/problems/${refusal}`,
title: TITLE[refusal],
status: STATUS[refusal],
code: refusal,
detail: error.message,
})
}
/** A refusal waiting cannot fix: the index job records it on the document instead of retrying it. */
export function permanentRefusal(error: KnowledgeError): KnowledgeRefusal | undefined {
const refusal = refusalOfReason(error.reason)
return refusal === "indexing" || refusal === "runtime_unavailable" ? undefined : refusal
}
+20
View File
@@ -0,0 +1,20 @@
// The live collection of the knowledge module (umbrella TS6.1). The live server announces a change of
// `knowledge_document` -- an index asked for, a job taking it, the platform answering indexed or failed, a
// removal -- as `{ entity: "knowledge.document", id }`, ids only; a reader reads the document back through
// `knowledge.document.getMany`, so the procedure's permission and the repository's tenant wall decide what a
// member sees. A topic is a project, so a page that shows one collection filters the rows by its name.
// `pnpm kit check` refuses a collection without its batch-get procedure.
import type { ModuleCollection } from "@conusai-kit/core"
export const DOCUMENT_COLLECTION = {
entity: "knowledge.document",
table: "knowledge_document",
key: "documentId",
scopeLevel: "project",
read: "knowledge.knowledge.search",
list: "knowledge.document.list",
getMany: "knowledge.document.getMany",
} as const satisfies ModuleCollection
export const KNOWLEDGE_COLLECTIONS: readonly ModuleCollection[] = [DOCUMENT_COLLECTION]
+50
View File
@@ -0,0 +1,50 @@
// The constants of the knowledge module: its feature flag, jobs, events, audit actions and the
// indexable content types.
/** The module flag: knowledge retrieval and indexing, off until an administrator enables document AI. */
export const KNOWLEDGE_FLAG = "knowledge.enabled"
export const KNOWLEDGE_FLAG_DEFAULTS = {
[KNOWLEDGE_FLAG]: false,
} as const
/** The worker job that hands one document to the platform's knowledge service. */
export const INDEX_JOB = "knowledge.index"
/** The worker job that removes one document from the platform's knowledge service. */
export const REMOVE_JOB = "knowledge.remove"
export const COLLECTION_CREATED = "knowledge.collection.created"
export const DOCUMENT_INDEX_REQUESTED = "knowledge.document.index_requested"
export const DOCUMENT_REMOVE_REQUESTED = "knowledge.document.remove_requested"
export const DOCUMENT_AI_ENABLED = "knowledge.settings.document_ai_enabled"
export const DOCUMENT_AI_DISABLED = "knowledge.settings.document_ai_disabled"
/** A project's automatic PDF indexing switched on (naming a collection) or off. */
export const AUTO_INDEX_CHANGED = "knowledge.autoindex.changed"
/** The worker job that reads one uploaded PDF and indexes its passages into the project's opted-in collection. */
export const AUTO_INDEX_JOB = "knowledge.autoindex"
/** The files processor that enqueues it on every upload of a matching type; also its jobs' singleton namespace. */
export const AUTO_INDEX_PROCESSOR_KEY = "knowledge.autoindex"
/** What the automatic indexing reads, through the platform's document reader. */
export const AUTO_INDEX_CONTENT_TYPES = ["application/pdf"] as const
/** A PDF's ceiling for the reader: the files module's upload ceiling, as the documents module reads one. */
export const MAX_AUTO_INDEX_BYTES = 50 * 1024 * 1024
/**
* What the indexer reads as text. PDFs and scans are the documents module's worked example; this
* module indexes text-shaped files, and refuses anything else at the door with a named field.
*/
export const INDEXABLE_CONTENT_TYPES = ["text/plain", "text/markdown", "application/json", "text/csv"] as const
/** A search query's ceiling; a retrieval question is a sentence, not a document. */
export const MAX_QUERY_LENGTH = 500
/** What one search returns at most, however much the caller asks (the retrieval limit contract). */
export const MAX_SEARCH_RESULTS = 25
/** An indexed file's ceiling: one megabyte of text is already far past useful chunking. */
export const MAX_INDEX_BYTES = 1_048_576
+105
View File
@@ -0,0 +1,105 @@
// The tables of the knowledge module. Five.
//
// `knowledge_collection` is one named collection of documents, project-owned; the collection name
// is the slug procedures take (a procedure's input carries slugs, never ids). `knowledge_document`
// is one file's indexing state in one collection — the row the UI's progress states read, and the
// idempotency key of the index job (a file is indexed into a collection once, and re-indexing the
// same pair resumes the one row). `knowledge_config` is the organization's enablement state: the
// durable answer to "is document AI on" and which backend serves it. `knowledge_auto_index` is a
// project's opt-in to indexing its uploaded PDFs, and `knowledge_passage_set` what one such PDF sent
// to the service, so removing it can tombstone each passage. No secret ever lands here.
import { defineEntity, idSchema, instantSchema } from "@conusai-kit/data"
import { z } from "zod"
import { DOCUMENT_STATUSES, KNOWLEDGE_BACKENDS } from "./statuses.js"
export const collections = defineEntity({
name: "collection",
table: "knowledge_collection",
tenancy: "project",
fields: {
name: z.string().min(1).max(100),
createdBy: z.string().min(1).max(128),
},
unique: [["name"]],
})
export const documents = defineEntity({
name: "document",
table: "knowledge_document",
tenancy: "project",
fields: {
collectionName: z.string().min(1).max(100),
/** The row of the files module this document indexes: an opaque id, never a foreign key. */
fileId: idSchema,
filename: z.string().min(1).max(255),
status: z.enum(DOCUMENT_STATUSES),
/** A stable failure code when the indexing failed; null otherwise. */
error: z.string().max(200).nullable(),
indexedAt: z.string().nullable(),
requestedBy: z.string().min(1).max(128),
},
// One row per (collection, file): a re-index resumes the same row instead of duplicating it.
unique: [["collectionName", "fileId"]],
indexes: [["collectionName"]],
})
export const config = defineEntity({
name: "config",
table: "knowledge_config",
tenancy: "org",
fields: {
enabled: z.boolean(),
/** Which search backend serves this organization. */
backend: z.enum(KNOWLEDGE_BACKENDS),
// The three below held the app's own vector store, which is gone (ADR-061); the migration that made
// them has run, so they stay as "managed, nothing to link" and nothing reads them.
source: z.enum(["managed"]),
serviceId: z.string().max(128).nullable(),
url: z.string().max(500).nullable(),
updatedBy: z.string().min(1).max(128),
/** When the configuration last changed; the managed updated_at is the row's own bookkeeping. */
configuredAt: instantSchema,
},
})
/**
* A project's opt-in to indexing its uploaded PDFs automatically (core-11): no row means off. It names
* the collection the files go into and the administrator who turned it on, whose permission every
* indexing job checks again, since the job runs for nobody else.
*/
export const autoIndex = defineEntity({
name: "autoindex",
table: "knowledge_auto_index",
tenancy: "project",
fields: {
collectionName: z.string().min(1).max(100),
enabledBy: z.string().min(1).max(128),
enabledAt: instantSchema,
},
})
/**
* The passages an automatic PDF index sent for one document (core-11). The service indexed one inline
* item per passage under the file's reference plus `#p{page}.{index}`, and removes a reference only
* exactly, so the row keeps the file's reference (which outlives the file) and the count of passages
* on each page, comma-separated from page 1 ("2,0,1": two on page 1, none on page 2, one on page 3).
*/
export const passageSets = defineEntity({
name: "passageset",
table: "knowledge_passage_set",
tenancy: "project",
fields: {
collectionName: z.string().min(1).max(100),
fileId: idSchema,
documentRef: z.string().min(1).max(1024),
pages: z.string().max(65_536),
},
unique: [["collectionName", "fileId"]],
})
/** The tables of the first migration, as first shipped. */
export const KNOWLEDGE_ENTITIES_V1 = Object.freeze([collections, documents, config])
/** Every table this module owns, in the order its migrations create them. */
export const KNOWLEDGE_ENTITIES = Object.freeze([...KNOWLEDGE_ENTITIES_V1, autoIndex, passageSets])
+41
View File
@@ -0,0 +1,41 @@
// The events of the knowledge module. A write procedure publishes the request; the module's own
// subscriber (running in the worker's outbox relay) enqueues the job, which is how a web request
// never waits on the platform's knowledge service.
import { defineEvent } from "@conusai-kit/core"
import { z } from "zod"
const inProject = {
orgId: z.string(),
workspaceId: z.string(),
projectId: z.string(),
}
/** A document was asked to be indexed: the subscriber enqueues the index job. */
export const documentIndexRequested = defineEvent(
"knowledge.documentIndexRequested",
z.object({
...inProject,
documentId: z.string(),
collectionName: z.string(),
fileId: z.string(),
}),
)
/**
* A document was asked to be removed: the subscriber enqueues the remove job. `storageKey` names the object to
* remove, resolved when the removal is asked for (from `FileDeleted`, or from the stored file a person removes),
* because the file may be gone when the job runs. It is optional only for a record written before it existed.
*/
export const documentRemoveRequested = defineEvent(
"knowledge.documentRemoveRequested",
z.object({
...inProject,
documentId: z.string(),
collectionName: z.string(),
fileId: z.string(),
storageKey: z.string().optional(),
}),
)
export const KNOWLEDGE_EVENTS = [documentIndexRequested, documentRemoveRequested]
+66
View File
@@ -0,0 +1,66 @@
// The public API of the knowledge module: what a page, another module and the OpenAPI document may
// import. Nothing here reaches core/, adapters/ or a driver.
export { DOCUMENT_COLLECTION, KNOWLEDGE_COLLECTIONS } from "./collections.js"
export {
AUTO_INDEX_CHANGED,
AUTO_INDEX_CONTENT_TYPES,
AUTO_INDEX_JOB,
AUTO_INDEX_PROCESSOR_KEY,
COLLECTION_CREATED,
DOCUMENT_AI_DISABLED,
DOCUMENT_AI_ENABLED,
DOCUMENT_INDEX_REQUESTED,
DOCUMENT_REMOVE_REQUESTED,
INDEX_JOB,
INDEXABLE_CONTENT_TYPES,
KNOWLEDGE_FLAG,
KNOWLEDGE_FLAG_DEFAULTS,
MAX_AUTO_INDEX_BYTES,
MAX_INDEX_BYTES,
MAX_QUERY_LENGTH,
MAX_SEARCH_RESULTS,
REMOVE_JOB,
} from "./constants.js"
export {
autoIndex,
collections,
config,
documents,
KNOWLEDGE_ENTITIES,
KNOWLEDGE_ENTITIES_V1,
passageSets,
} from "./entities.js"
export { documentIndexRequested, documentRemoveRequested, KNOWLEDGE_EVENTS } from "./events.js"
export type { CollectionView, DocumentView, SearchResult } from "./procedures.js"
export {
autoIndexGet,
autoIndexSet,
collectionCreate,
collectionList,
collectionName,
collectionViewSchema,
documentGetMany,
documentIndex,
documentList,
documentRemove,
documentViewSchema,
KNOWLEDGE_PROCEDURES,
knowledgeSearch,
knowledgeStatus,
searchResultSchema,
settingsDisableDocumentAi,
settingsEnableDocumentAi,
settingsGet,
} from "./procedures.js"
export type { DocumentStatus, EnablementReason, IndexFailure } from "./statuses.js"
export {
ENABLEMENT_REASONS,
INDEX_FAILURES,
KNOWLEDGE_BACKENDS,
KNOWLEDGE_REFUSALS,
type KnowledgeBackend,
type KnowledgeRefusal,
SERVICE_STATES,
type ServiceState,
} from "./statuses.js"
+229
View File
@@ -0,0 +1,229 @@
// The procedure contracts of the knowledge module: schemas and names, no handler and no port.
//
// The module is a retrieval layer over the app's documents: a collection of text files is indexed
// (in the worker: the platform's knowledge service reads, chunks and embeds it), searched
// with cited snippets, and managed through an explicit, idempotent enablement flow. The enablement
// procedure is named enableDocumentAi per the Document AI enablement contract; its permission is the module's
// manage permission, so the admin gate is one line in the manifest.
import { procedure } from "@conusai-kit/core"
import { z } from "zod"
import { AUTO_INDEX_CHANGED, COLLECTION_CREATED, DOCUMENT_AI_DISABLED, DOCUMENT_AI_ENABLED } from "./constants.js"
import { type DocumentStatus, KNOWLEDGE_BACKENDS, KNOWLEDGE_REFUSALS, type ServiceState } from "./statuses.js"
const inProject = { org: z.string(), workspace: z.string(), project: z.string() }
const inOrg = { org: z.string() }
export const collectionName = z
.string()
.min(1)
.max(100)
.regex(/^[a-z][a-z0-9-]*$/, "lower-case letters, digits and dashes")
export const collectionViewSchema = z.object({
name: collectionName,
documentCount: z.number().int(),
indexedCount: z.number().int(),
createdBy: z.string(),
})
export type CollectionView = z.output<typeof collectionViewSchema>
export const documentViewSchema = z.object({
documentId: z.string(),
collectionName: collectionName,
fileId: z.string(),
filename: z.string(),
status: z.enum(["queued", "indexing", "indexed", "failed", "cancelled"]),
error: z.string().nullable(),
indexedAt: z.string().nullable(),
requestedBy: z.string(),
})
export type DocumentView = z.output<typeof documentViewSchema>
export const searchResultSchema = z.object({
fileId: z.string(),
filename: z.string(),
collectionName: collectionName,
snippet: z.string(),
page: z.number().int().min(1).optional(),
blockId: z.string().optional(),
chunkId: z.string().optional(),
/** 0..1; higher is closer. */
score: z.number(),
/** Result ordinal only. Page/block/chunkId carry actual returned source locators when available. */
chunk: z.number().int().min(1),
})
export type SearchResult = z.output<typeof searchResultSchema>
export const collectionList = procedure("knowledge.collection.list")
.kind("read")
.scope("project")
.permission("knowledge.knowledge.search")
.input(z.object({ ...inProject }))
.output(z.object({ collections: z.array(collectionViewSchema) }))
.contract()
export const collectionCreate = procedure("knowledge.collection.create")
.kind("write")
.scope("project")
.permission("knowledge.collection.manage")
.input(z.object({ ...inProject, name: collectionName }))
.output(collectionViewSchema)
.audit(COLLECTION_CREATED)
.contract()
export const documentIndex = procedure("knowledge.document.index")
.kind("write")
.scope("project")
.permission("knowledge.knowledge.manage")
.input(z.object({ ...inProject, collectionName: collectionName, fileId: z.string() }))
.output(documentViewSchema)
.contract()
export const documentList = procedure("knowledge.document.list")
.kind("read")
.scope("project")
.permission("knowledge.knowledge.search")
.input(z.object({ ...inProject, collectionName: collectionName }))
.output(z.object({ documents: z.array(documentViewSchema) }))
.contract()
/**
* The live collection's batch-get (contract C-05): the documents with these ids, in whichever collection of
* the project, under the list's permission. An id of another tenant, of another project or of nothing is
* left out of the answer.
*/
export const documentGetMany = procedure("knowledge.document.getMany")
.kind("read")
.scope("project")
.permission("knowledge.knowledge.search")
.input(z.object({ ...inProject, ids: z.array(z.string().min(1).max(64)).min(1).max(100) }))
.output(z.object({ items: z.array(documentViewSchema) }))
.contract()
export const documentRemove = procedure("knowledge.document.remove")
.kind("write")
.scope("project")
.permission("knowledge.knowledge.manage")
.input(z.object({ ...inProject, collectionName: collectionName, fileId: z.string() }))
.output(documentViewSchema)
.contract()
/**
* The retrieval call. It embeds the query through the AI gateway before it searches, so it is an
* effect: a read procedure cannot leave the process. Exposed as an MCP tool on purpose (`.tool()`).
*/
export const knowledgeSearch = procedure("knowledge.search")
.kind("effect")
.scope("project")
.permission("knowledge.knowledge.search")
.tool()
.input(
z.object({
...inProject,
collectionName: collectionName,
query: z.string().min(1).max(500),
limit: z.number().int().min(1).max(25).optional(),
}),
)
.output(z.object({ results: z.array(searchResultSchema) }))
.contract()
export const knowledgeStatus = procedure("knowledge.status")
.kind("read")
.scope("project")
.permission("knowledge.knowledge.search")
.input(z.object({ ...inProject }))
.output(
z.object({
/** The module flag's answer for this project. */
enabled: z.boolean(),
/** The service readiness, kept separate from the indexing indicator below. */
service: z.object({
state: z.enum(["registered", "indexing", "ready", "degraded", "disabled", "unavailable"]),
reasons: z.array(z.string()),
/** Why the service will not serve this deployment, when that is the answer: the page's own state. */
refusal: z.enum(KNOWLEDGE_REFUSALS).optional(),
}),
documents: z.object({
queued: z.number().int(),
indexing: z.number().int(),
indexed: z.number().int(),
failed: z.number().int(),
cancelled: z.number().int(),
}),
}),
)
.contract()
export const settingsGet = procedure("knowledge.settings.get")
.kind("read")
.scope("org")
.permission("knowledge.settings.manage")
.input(z.object({ ...inOrg }))
.output(
z.object({
enabled: z.boolean(),
backend: z.enum(KNOWLEDGE_BACKENDS).nullable(),
}),
)
.contract()
/** Its checks and the managed link call the platform's APIs, so it is an effect, like the search. */
export const settingsEnableDocumentAi = procedure("knowledge.settings.enableDocumentAi")
.kind("effect")
.scope("org")
.permission("knowledge.settings.manage")
.input(z.object({ ...inOrg }))
.output(z.object({ enabled: z.boolean(), backend: z.enum(KNOWLEDGE_BACKENDS) }))
.audit(DOCUMENT_AI_ENABLED)
.contract()
export const settingsDisableDocumentAi = procedure("knowledge.settings.disableDocumentAi")
.kind("write")
.scope("org")
.permission("knowledge.settings.manage")
.input(z.object({ ...inOrg }))
.output(z.object({ enabled: z.boolean() }))
.audit(DOCUMENT_AI_DISABLED)
.contract()
/** Whether this project indexes its uploaded PDFs automatically, and into which collection. */
export const autoIndexGet = procedure("knowledge.autoindex.get")
.kind("read")
.scope("project")
.permission("knowledge.knowledge.search")
.input(z.object({ ...inProject }))
.output(z.object({ collectionName: collectionName.nullable() }))
.contract()
/**
* Turns automatic PDF indexing on (into an existing collection) or off (`null`). The administrator who
* turns it on is the one every indexing job re-checks: their permission is the job's authority.
*/
export const autoIndexSet = procedure("knowledge.autoindex.set")
.kind("write")
.scope("project")
.permission("knowledge.knowledge.manage")
.input(z.object({ ...inProject, collectionName: collectionName.nullable() }))
.output(z.object({ collectionName: collectionName.nullable() }))
.audit(AUTO_INDEX_CHANGED)
.contract()
export const KNOWLEDGE_PROCEDURES = [
collectionList,
collectionCreate,
documentIndex,
documentList,
documentGetMany,
documentRemove,
knowledgeSearch,
knowledgeStatus,
settingsGet,
settingsEnableDocumentAi,
settingsDisableDocumentAi,
autoIndexGet,
autoIndexSet,
]
export type { DocumentStatus, ServiceState }
+53
View File
@@ -0,0 +1,53 @@
// The statuses a knowledge document moves through. `unavailable` and `disabled` are not rows: they
// are the module's and the service's answers, and the UI maps all eight states from what a page
// reads (the module flag, the service readiness, the document row).
export const DOCUMENT_STATUSES = ["queued", "indexing", "indexed", "failed", "cancelled"] as const
export type DocumentStatus = (typeof DOCUMENT_STATUSES)[number]
/** The search backend a deployment binds: the platform's managed service, and no other (ADR-061). */
export const KNOWLEDGE_BACKENDS = ["platform"] as const
export type KnowledgeBackend = (typeof KNOWLEDGE_BACKENDS)[number]
/** The readiness states the search port answers (the managed toolset's readiness + unavailable). */
export const SERVICE_STATES = ["registered", "indexing", "ready", "degraded", "disabled", "unavailable"] as const
export type ServiceState = (typeof SERVICE_STATES)[number]
/** The enablement flow's typed refusal reasons, surfaced in the UI as states. */
export const ENABLEMENT_REASONS = ["plan_required", "feature_disabled", "quota_exhausted"] as const
export type EnablementReason = (typeof ENABLEMENT_REASONS)[number]
/**
* Why the platform's knowledge service will not serve this deployment, as the pages show it. The service's
* reason codes reach the module as these (adapters/refusals.ts): `scope_missing` is the service refusing
* the deployment's token, whose `conusai.json` asks for `knowledge.collections:use`.
*/
export const KNOWLEDGE_REFUSALS = [
"plan_required",
"feature_disabled",
"quota_exhausted",
"budget_exhausted",
"scope_missing",
"indexing",
"approval_required",
"runtime_unavailable",
"configuration_missing",
] as const
export type KnowledgeRefusal = (typeof KNOWLEDGE_REFUSALS)[number]
/** The indexing failure codes a document row records: the file's own, and a refusal waiting cannot fix. */
export const INDEX_FAILURES = [
"processing_exhausted",
"configuration_missing",
"not_indexable",
"too_large",
"not_text",
"file_changed",
"plan_required",
"feature_disabled",
"quota_exhausted",
"budget_exhausted",
"scope_missing",
"approval_required",
] as const
export type IndexFailure = (typeof INDEX_FAILURES)[number]
+72
View File
@@ -0,0 +1,72 @@
// The automatic PDF intake (template lesson core-11): what the worker runs for one uploaded file the
// files module handed to this module's processor. A project that opted in (`knowledge_auto_index`)
// gets the PDF indexed into its collection as a document like any other, requested by the
// administrator who opted in: the job runs for nobody else, so that administrator's admission is what
// each step leaving the process checks again (`beforeIndex`). Anything else reads nothing.
import { AppError } from "@conusai-kit/core"
import { getFile } from "@conusai-kit/files"
import { AUTO_INDEX_CONTENT_TYPES, autoIndex, type DocumentView, documents } from "../contract/index.js"
import { indexPassages } from "./index-job.js"
import { type KnowledgeDeps, type KnowledgeScope, knowledgeEnabled } from "./ports.js"
/** What the files module's processor job carries about the upload. */
export interface AutoIndexUpload {
fileId: string
contentType: string
sha256: string
}
/**
* Indexes one upload of an opted-in project, or answers null and reads nothing: a type it does not read,
* a project that is not opted in or has knowledge off, a file that is gone, or one whose digest is not
* the upload's. A redelivered job finds the document row and resumes it, so the work is done once.
*/
export async function autoIndexUpload(
deps: KnowledgeDeps,
scope: KnowledgeScope,
upload: AutoIndexUpload,
options: { signal?: AbortSignal; finalAttempt?: boolean } = {},
): Promise<DocumentView | null> {
if (!(AUTO_INDEX_CONTENT_TYPES as readonly string[]).includes(upload.contentType)) return null
const optIn = (await deps.data.forScope(scope).of(autoIndex).list({ limit: 1 })).items[0]
if (!optIn) return null
if (!(await knowledgeEnabled(deps, scope))) return null
const file = await storedFile(deps, scope, upload.fileId)
if (!file || file.sha256 !== upload.sha256) return null
const row = await deps.data.unitOfWork.run(scope, async (tx) => {
const existing = (
await tx.of(documents).list({
where: [
{ field: "collectionName", op: "eq", value: optIn.collectionName },
{ field: "fileId", op: "eq", value: upload.fileId },
],
limit: 1,
})
).items[0]
return (
existing ??
(await tx.of(documents).insert({
collectionName: optIn.collectionName,
fileId: upload.fileId,
filename: file.filename,
status: "queued",
error: null,
indexedAt: null,
requestedBy: optIn.enabledBy,
}))
)
})
return indexPassages(deps, scope, row.id, options)
}
/** The upload's file, or undefined once it is no longer stored (deleted, or on its way out). */
async function storedFile(deps: KnowledgeDeps, scope: KnowledgeScope, fileId: string) {
try {
return await getFile(deps.files, scope, fileId)
} catch (error) {
if (error instanceof AppError && error.status === 404) return undefined
throw error
}
}
+46
View File
@@ -0,0 +1,46 @@
// A project's automatic PDF indexing (template lesson core-11): an administrator opts the project in,
// naming the collection its uploaded PDFs go into; no row is off. The administrator is recorded,
// because the indexing job runs for nobody and re-checks that person's permission before each file.
import { forbidden } from "@conusai-kit/core"
import { autoIndex } from "../contract/index.js"
import { requireCollection } from "./collections.js"
import { type KnowledgeDeps, type KnowledgeScope, knowledgeEnabled } from "./ports.js"
export interface AutoIndexView {
/** The collection uploaded PDFs are indexed into, or null when automatic indexing is off. */
collectionName: string | null
}
export async function getAutoIndex(deps: Pick<KnowledgeDeps, "data">, scope: KnowledgeScope): Promise<AutoIndexView> {
const row = (await deps.data.forScope(scope).of(autoIndex).list({ limit: 1 })).items[0]
return { collectionName: row?.collectionName ?? null }
}
/** Turns automatic indexing on into `collectionName` (which must exist), or off with `null`. */
export async function setAutoIndex(
deps: KnowledgeDeps & { now?: () => number },
scope: KnowledgeScope,
collectionName: string | null,
): Promise<AutoIndexView> {
if (collectionName !== null) {
// The tenant-wall read first: another project's collection answers NotFound, never a policy refusal.
await requireCollection(deps, scope, collectionName)
if (!(await knowledgeEnabled(deps, scope))) throw forbidden("knowledge is switched off for this project")
}
await deps.data.unitOfWork.run(scope, async (tx) => {
const row = (await tx.of(autoIndex).list({ limit: 1 })).items[0]
if (collectionName === null) {
if (row) await tx.of(autoIndex).delete(row.id)
return
}
const patch = {
collectionName,
enabledBy: scope.actor.userId,
enabledAt: new Date((deps.now ?? Date.now)()).toISOString(),
}
if (row) await tx.of(autoIndex).update(row.id, patch, row.version)
else await tx.of(autoIndex).insert(patch)
})
return { collectionName }
}
+132
View File
@@ -0,0 +1,132 @@
// The collections: list them with their document counts, create one. A collection is a named
// bucket; its name is the slug every procedure takes, so a page never handles an id.
import { conflict, notFound, validationFailed } from "@conusai-kit/core"
import { type CollectionView, collections, documents } from "../contract/index.js"
import type { KnowledgeDeps, KnowledgeScope } from "./ports.js"
const PAGE = 200
export async function listCollections(
deps: KnowledgeDeps,
scope: KnowledgeScope,
): Promise<{ collections: CollectionView[] }> {
const found = []
let cursor: string | undefined
for (;;) {
const page = await deps.data
.forScope(scope)
.of(collections)
.list({
order: [{ field: "createdAt", direction: "asc" }],
limit: PAGE,
...(cursor === undefined ? {} : { cursor }),
})
found.push(...page.items)
if (!page.hasMore) break
cursor = page.cursor
}
const views: CollectionView[] = []
for (const row of found) {
const counts = await documentCounts(deps, scope, row.name)
views.push({
name: row.name,
documentCount: counts.total,
indexedCount: counts.indexed,
createdBy: row.createdBy,
})
}
return { collections: views }
}
async function documentCounts(
deps: KnowledgeDeps,
scope: KnowledgeScope,
collectionName: string,
): Promise<{ total: number; indexed: number }> {
let total = 0
let indexed = 0
let cursor: string | undefined
for (;;) {
const page = await deps.data
.forScope(scope)
.of(documents)
.list({
where: [{ field: "collectionName", op: "eq", value: collectionName }],
limit: PAGE,
...(cursor === undefined ? {} : { cursor }),
})
for (const row of page.items) {
total += 1
if (row.status === "indexed") indexed += 1
}
if (!page.hasMore) break
cursor = page.cursor
}
return { total, indexed }
}
/**
* Creates a collection, idempotently: a name that already exists answers with the existing
* collection and writes nothing, so a retried request never audits twice.
*/
export async function createCollection(
deps: KnowledgeDeps,
scope: KnowledgeScope,
name: string,
): Promise<CollectionView> {
const repos = deps.data.forScope(scope)
const existing = (await repos.of(collections).list({ where: [{ field: "name", op: "eq", value: name }], limit: 1 }))
.items[0]
if (existing) {
const counts = await documentCounts(deps, scope, name)
return {
name: existing.name,
documentCount: counts.total,
indexedCount: counts.indexed,
createdBy: existing.createdBy,
}
}
try {
const row = await deps.data.unitOfWork.run(scope, async (tx) => {
const inserted = await tx.of(collections).insert({ name, createdBy: scope.actor.userId })
return inserted
})
return { name: row.name, documentCount: 0, indexedCount: 0, createdBy: row.createdBy }
} catch (error) {
// A concurrent create with the same name lost the race: answer the winner's row.
if ((error as { status?: number }).status === 409) {
const winner = (await repos.of(collections).list({ where: [{ field: "name", op: "eq", value: name }], limit: 1 }))
.items[0]
if (winner) {
const counts = await documentCounts(deps, scope, name)
return {
name: winner.name,
documentCount: counts.total,
indexedCount: counts.indexed,
createdBy: winner.createdBy,
}
}
}
throw error
}
}
/** One collection by name, or NotFound: the resolver's own read is the membership check. */
export async function requireCollection(deps: KnowledgeDeps, scope: KnowledgeScope, name: string) {
const row = (
await deps.data
.forScope(scope)
.of(collections)
.list({ where: [{ field: "name", op: "eq", value: name }], limit: 1 })
).items[0]
if (!row) throw notFound("no such collection")
return row
}
export const unknownCollection = (name: string): never => {
throw validationFailed([{ path: "collectionName", message: "no such collection" }], `no collection ${name}`)
}
export const takenCollection = (): never => {
throw conflict("this collection name is taken")
}
+255
View File
@@ -0,0 +1,255 @@
// The documents: request an index (a web write that records the row and publishes the outbox
// event the worker's subscriber turns into the index job), request a removal the same way, and
// list a collection's documents with their progress states.
import { AppError, conflict, forbidden, notFound, validationFailed } from "@conusai-kit/core"
import { getFile, getFileStorage } from "@conusai-kit/files"
import {
type DocumentView,
documentIndexRequested,
documentRemoveRequested,
documents,
INDEXABLE_CONTENT_TYPES,
MAX_INDEX_BYTES,
} from "../contract/index.js"
import { requireCollection } from "./collections.js"
import { type KnowledgeDeps, type KnowledgeScope, knowledgeEnabled } from "./ports.js"
const PAGE = 200
export const documentView = (row: {
id: string
collectionName: string
fileId: string
filename: string
status: DocumentView["status"]
error: string | null
indexedAt: string | null
requestedBy: string
}): DocumentView => ({
documentId: row.id,
collectionName: row.collectionName,
fileId: row.fileId,
filename: row.filename,
status: row.status,
error: row.error,
indexedAt: row.indexedAt,
requestedBy: row.requestedBy,
})
export async function listDocuments(
deps: KnowledgeDeps,
scope: KnowledgeScope,
collectionName: string,
): Promise<{ documents: DocumentView[] }> {
const found = []
let cursor: string | undefined
for (;;) {
const page = await deps.data
.forScope(scope)
.of(documents)
.list({
where: [{ field: "collectionName", op: "eq", value: collectionName }],
order: [{ field: "createdAt", direction: "desc" }],
limit: PAGE,
...(cursor === undefined ? {} : { cursor }),
})
found.push(...page.items)
if (!page.hasMore) break
cursor = page.cursor
}
return { documents: found.map((row) => documentView(row)) }
}
/** The most ids one batch-get reads (contract C-05). */
export const MAX_BATCH = 100
/**
* The live collection's read-back: the documents with these ids, in whichever collection of the project.
* A row this caller cannot see is not in the answer: the repository's tenant wall leaves it out.
*/
export async function getManyDocuments(
deps: KnowledgeDeps,
scope: KnowledgeScope,
documentIds: readonly string[],
): Promise<DocumentView[]> {
const page = await deps.data
.forScope(scope)
.of(documents)
.list({ where: [{ field: "id", op: "in", values: [...new Set(documentIds)] }], limit: MAX_BATCH })
return page.items.map((row) => documentView(row))
}
/**
* Asks for a document's indexing. The write is three things in one transaction: the durable row
* (the UI's progress states and the job's idempotency key), the outbox event the worker's
* subscriber enqueues the index job from, and nothing else — the call to the platform's knowledge
* service happens in the worker, never in a request (AGENTS.md: long work is a job).
*
* Idempotent by (collection, file): asking again for a pair that is queued or indexing answers
* the same row and publishes nothing; asking for an indexed or failed pair re-queues the one row,
* so a retry can never duplicate a document.
*/
export async function requestIndex(
deps: KnowledgeDeps,
scope: KnowledgeScope,
input: { collectionName: string; fileId: string },
): Promise<DocumentView> {
// The tenant-wall reads come before the flag check: another tenant's ids answer NotFound, never
// a policy refusal, so ids do not leak existence behind a disabled flag.
await requireCollection(deps, scope, input.collectionName)
const file = await getFile(deps.files, scope, input.fileId)
if (!(INDEXABLE_CONTENT_TYPES as readonly string[]).includes(file.contentType)) {
throw validationFailed(
[{ path: "fileId", message: `${file.contentType} cannot be indexed; index a text file` }],
"this file type is not indexable",
)
}
if (file.size > MAX_INDEX_BYTES) {
throw validationFailed(
[{ path: "fileId", message: `the file is larger than ${MAX_INDEX_BYTES} bytes` }],
"the file is too large to index",
)
}
if (!(await knowledgeEnabled(deps, scope))) throw forbidden("knowledge is switched off for this project")
return deps.data.unitOfWork.run(scope, async (tx) => {
const existing = (
await tx.of(documents).list({
where: [
{ field: "collectionName", op: "eq", value: input.collectionName },
{ field: "fileId", op: "eq", value: input.fileId },
],
limit: 1,
})
).items[0]
if (existing && (existing.status === "queued" || existing.status === "indexing")) {
return documentView(existing)
}
const row =
existing !== undefined
? await tx
.of(documents)
.update(
existing.id,
{ status: "queued", error: null, indexedAt: null, requestedBy: scope.actor.userId },
existing.version,
)
: await tx.of(documents).insert({
collectionName: input.collectionName,
fileId: input.fileId,
filename: file.filename,
status: "queued",
error: null,
indexedAt: null,
requestedBy: scope.actor.userId,
})
await tx.publish(documentIndexRequested, {
orgId: scope.orgId,
workspaceId: scope.workspaceId,
projectId: scope.projectId,
documentId: row.id,
collectionName: input.collectionName,
fileId: input.fileId,
})
return documentView(row)
})
}
/** Asks for a document's removal: the row and the event in one transaction, the job in the worker. */
export async function requestRemove(
deps: KnowledgeDeps,
scope: KnowledgeScope,
input: { collectionName: string; fileId: string; storageKey?: string },
): Promise<DocumentView> {
const current = (
await deps.data
.forScope(scope)
.of(documents)
.list({
where: [
{ field: "collectionName", op: "eq", value: input.collectionName },
{ field: "fileId", op: "eq", value: input.fileId },
],
limit: 1,
})
).items[0]
// The tenant-wall read comes first, as in requestIndex: another tenant's ids answer NotFound.
if (!current) throw notFound("no such document")
if (current.status === "cancelled") return documentView(current)
// Every removal names the object it removes, resolved when it is asked for, not when the job runs: by then the
// file may be gone (F407). A deletion brings the key it had; a person's removal reads it from the stored file.
const storageKey = input.storageKey ?? (await storageKeyOf(deps, scope, input.fileId))
return deps.data.unitOfWork.run(scope, async (tx) => {
const row = (
await tx.of(documents).list({
where: [
{ field: "collectionName", op: "eq", value: input.collectionName },
{ field: "fileId", op: "eq", value: input.fileId },
],
limit: 1,
})
).items[0]
if (!row) throw notFound("no such document")
if (row.status === "cancelled") return documentView(row)
const next = await tx.of(documents).update(row.id, { status: "cancelled", error: null }, row.version)
await tx.publish(documentRemoveRequested, {
orgId: scope.orgId,
workspaceId: scope.workspaceId,
projectId: scope.projectId,
documentId: row.id,
collectionName: input.collectionName,
fileId: input.fileId,
storageKey,
})
return documentView(next)
})
}
/** The document row a job resumes by; NotFound when another tenant's id was named. */
export async function documentById(deps: KnowledgeDeps, scope: KnowledgeScope, documentId: string) {
const row = await deps.data.forScope(scope).of(documents).get(documentId)
if (!row) throw notFound("no such document")
return row
}
/**
* The stored file's object key. A file on its way out (being deleted, or gone) is refused rather than removed
* unkeyed: its deletion publishes `FileDeleted` with the key, and that removal finds the document still active.
*/
async function storageKeyOf(deps: KnowledgeDeps, scope: KnowledgeScope, fileId: string): Promise<string> {
try {
return (await getFileStorage(deps.files, scope, fileId)).storageKey
} catch (error) {
if (error instanceof AppError && error.status === 404) {
throw conflict("the file is being deleted; its indexed text is removed with it")
}
throw error
}
}
/** A source deletion revokes every collection's citation and durably requests cleanup.
* Does not require processing enablement: deleting retained data is independent of inference.
* `storageKey` is the key the deleted file had (`FileDeleted`): its row is gone, so the removal cannot look it up. */
export async function requestSourceRemoval(
deps: KnowledgeDeps,
scope: KnowledgeScope,
fileId: string,
storageKey: string,
): Promise<void> {
let cursor: string | undefined
for (;;) {
const page = await deps.data
.forScope(scope)
.of(documents)
.list({
where: [{ field: "fileId", op: "eq", value: fileId }],
limit: PAGE,
...(cursor === undefined ? {} : { cursor }),
})
for (const row of page.items)
await requestRemove(deps, scope, { collectionName: row.collectionName, fileId, storageKey })
if (!page.hasMore) break
cursor = page.cursor
}
}
+280
View File
@@ -0,0 +1,280 @@
// The index job's body: what the worker runs for one document. Read the file's bytes, hand them to
// the search service, and record the outcome on the durable row.
// Transient failures (the service hiccups) throw, and the queue retries the job; the row the
// retry resumes from makes the retry safe. Permanent failures are recorded on the row with a
// stable code, so a person sees why a document failed and a retried request does not pay twice.
// The automatic PDF intake (core-11) runs the same lifecycle with another send: the PDF is read into
// passages and each passage is indexed on its own (`passageSend`).
import { internal } from "@conusai-kit/core"
import { downloadFile, getFile, readAll, sha256Hex } from "@conusai-kit/files"
import {
type DocumentView,
documents,
INDEX_FAILURES,
type IndexFailure,
MAX_AUTO_INDEX_BYTES,
MAX_INDEX_BYTES,
passageSets,
} from "../contract/index.js"
import { documentById, documentView } from "./documents.js"
import { type KnowledgeDeps, type KnowledgePassage, type KnowledgeScope, knowledgeEnabled } from "./ports.js"
type DocumentRow = Awaited<ReturnType<typeof documentById>>
/** What one run sends between the row's "indexing" and its outcome; it throws to fail or to retry. */
type Send = (row: DocumentRow, signal: AbortSignal | undefined) => Promise<void>
export async function indexDocument(
deps: KnowledgeDeps,
scope: KnowledgeScope,
documentId: string,
options: { signal?: AbortSignal; finalAttempt?: boolean } = {},
): Promise<DocumentView> {
return runIndex(deps, scope, documentId, () => textSend(deps, scope), options)
}
/** The automatic PDF intake's run of one document: the PDF read into passages, each indexed on its own. */
export async function indexPassages(
deps: KnowledgeDeps,
scope: KnowledgeScope,
documentId: string,
options: { signal?: AbortSignal; finalAttempt?: boolean } = {},
): Promise<DocumentView> {
return runIndex(deps, scope, documentId, () => passageSend(deps, scope), options)
}
/** The whole file as text, to the search service. */
function textSend(deps: KnowledgeDeps, scope: KnowledgeScope): Send {
const service = deps.search
if (!service) throw internal("the knowledge search service is not bound in this process")
return async (row, signal) => {
const bytes = await readFileBytes(deps, scope, row.fileId, MAX_INDEX_BYTES)
const text = decodeText(bytes)
await deps.beforeIndex?.(row.requestedBy)
signal?.throwIfAborted()
await service.index(scope, {
collectionName: row.collectionName,
fileId: row.fileId,
filename: row.filename,
bytes: new TextEncoder().encode(text),
...(signal ? { signal } : {}),
})
}
}
/**
* A PDF, read by the platform's document reader into passages, each indexed as one item. What is about to
* be indexed is recorded before any of it is sent, so a run the service refuses partway can still be
* removed passage by passage. The read is keyed by the file and its digest: a retry reads the same job.
*/
function passageSend(deps: KnowledgeDeps, scope: KnowledgeScope): Send {
const { reader, passages, enablement } = deps
if (!reader || !passages) throw internal("the document reader and the passage index are not bound in this process")
if (!enablement) throw internal("the enablement platform is not bound in this process")
return async (row, signal) => {
const file = await getFile(deps.files, scope, row.fileId)
if (file.size > MAX_AUTO_INDEX_BYTES)
throw Object.assign(new Error("the file is too large to read"), { permanent: "too_large" })
// The organization's eligibility and quota, as the enablement flow checks them, before the read is paid
// for: a rollout kill switch or a spent quota stops the next PDF (root's core-11 decision). The service
// still refuses past its quota at call time; this check only stops earlier.
const eligibility = await enablement.checkEligibility(scope)
if (!eligibility.ok)
throw Object.assign(new Error(eligibility.detail ?? eligibility.reason), { permanent: eligibility.reason })
if ((await enablement.indexingQuota(scope)) <= 0)
throw Object.assign(new Error("the indexing volume for today is spent"), { permanent: "quota_exhausted" })
signal?.throwIfAborted()
const bytes = await readFileBytes(deps, scope, row.fileId, MAX_AUTO_INDEX_BYTES)
await deps.beforeIndex?.(row.requestedBy)
signal?.throwIfAborted()
const read = await reader.read(bytes, {
filename: row.filename,
idempotencyKey: `knowledge.autoindex.${row.fileId}.${file.sha256}`,
...(signal ? { signal } : {}),
})
const documentRef = await passages.reference(scope, row.fileId)
await recordPassages(deps, scope, row, documentRef, read)
await deps.beforeIndex?.(row.requestedBy)
signal?.throwIfAborted()
await passages.index(scope, {
collectionName: row.collectionName,
documentRef,
fileId: row.fileId,
passages: read,
...(signal ? { signal } : {}),
})
}
}
/** The passages per page, comma-separated from page 1: what `passagesOf` turns back into references. */
const pagesOf = (read: readonly KnowledgePassage[]): string => {
const counts: number[] = []
for (const passage of read) {
while (counts.length < passage.page) counts.push(0)
counts[passage.page - 1] = (counts[passage.page - 1] ?? 0) + 1
}
return counts.join(",")
}
const passagesOf = (pages: string): Array<{ page: number; index: number }> =>
pages === ""
? []
: pages
.split(",")
.flatMap((count, at) => Array.from({ length: Number(count) }, (_, index) => ({ page: at + 1, index })))
async function recordPassages(
deps: KnowledgeDeps,
scope: KnowledgeScope,
row: DocumentRow,
documentRef: string,
read: readonly KnowledgePassage[],
): Promise<void> {
const fields = { documentRef, pages: pagesOf(read) }
await deps.data.unitOfWork.run(scope, async (tx) => {
const existing = (await tx.of(passageSets).list(passageSetOf(row))).items[0]
if (existing) await tx.of(passageSets).update(existing.id, fields, existing.version)
else await tx.of(passageSets).insert({ collectionName: row.collectionName, fileId: row.fileId, ...fields })
})
}
/** The query of a document's passage record: one per (collection, file), as the document row. */
const passageSetOf = (row: { collectionName: string; fileId: string }) => ({
where: [
{ field: "collectionName" as const, op: "eq" as const, value: row.collectionName },
{ field: "fileId" as const, op: "eq" as const, value: row.fileId },
],
limit: 1,
})
async function runIndex(
deps: KnowledgeDeps,
scope: KnowledgeScope,
documentId: string,
sendOf: () => Send,
options: { signal?: AbortSignal; finalAttempt?: boolean },
): Promise<DocumentView> {
const row = await documentById(deps, scope, documentId)
if (row.status === "indexed" || row.status === "cancelled" || row.status === "failed") {
// A redelivered job or a crashed retry: the row says where the work stands.
return {
documentId: row.id,
collectionName: row.collectionName,
fileId: row.fileId,
filename: row.filename,
status: row.status,
error: row.error,
indexedAt: row.indexedAt,
requestedBy: row.requestedBy,
}
}
const send = sendOf()
const indexing = await deps.data
.forScope(scope)
.of(documents)
.update(row.id, { status: "indexing", error: null }, row.version)
try {
options.signal?.throwIfAborted()
if (!(await knowledgeEnabled(deps, scope)))
throw Object.assign(new Error("processing disabled"), { permanent: "feature_disabled" })
await deps.beforeIndex?.(row.requestedBy)
options.signal?.throwIfAborted()
await send(indexing, options.signal)
options.signal?.throwIfAborted()
const current = await deps.data.forScope(scope).of(documents).get(indexing.id)
if (!current || current.status === "cancelled") return documentView({ ...indexing, status: "cancelled" })
const done = await deps.data
.forScope(scope)
.of(documents)
.update(indexing.id, { status: "indexed", indexedAt: new Date().toISOString(), error: null }, indexing.version)
return {
documentId: done.id,
collectionName: done.collectionName,
fileId: done.fileId,
filename: done.filename,
status: done.status as DocumentView["status"],
error: done.error,
indexedAt: done.indexedAt,
requestedBy: done.requestedBy,
}
} catch (error) {
if (options.signal?.aborted) throw error
const permanent = permanentFailure(error) ?? (options.finalAttempt === true ? "processing_exhausted" : null)
if (permanent === null) throw error
const current = await deps.data.forScope(scope).of(documents).get(indexing.id)
if (!current || current.status === "cancelled") return documentView({ ...indexing, status: "cancelled" })
const failed = await deps.data
.forScope(scope)
.of(documents)
.update(indexing.id, { status: "failed", error: permanent }, indexing.version)
return {
documentId: failed.id,
collectionName: failed.collectionName,
fileId: failed.fileId,
filename: failed.filename,
status: "failed",
error: failed.error,
indexedAt: failed.indexedAt,
requestedBy: failed.requestedBy,
}
}
}
export async function removeDocument(
deps: KnowledgeDeps,
scope: KnowledgeScope,
documentId: string,
storageKey?: string,
): Promise<void> {
const row = await deps.data.forScope(scope).of(documents).get(documentId)
if (!row) return
const set = (await deps.data.forScope(scope).of(passageSets).list(passageSetOf(row))).items[0]
if (set) {
if (!deps.passages) throw internal("the passage index is not bound in this process")
await deps.passages.remove(scope, { documentRef: set.documentRef, passages: passagesOf(set.pages) })
await deps.data.unitOfWork.run(scope, async (tx) => {
await tx.of(documents).delete(row.id)
await tx.of(passageSets).delete(set.id)
})
return
}
const service = deps.search
if (!service) throw internal("the knowledge search service is not bound in this process")
await service.remove(scope, {
collectionName: row.collectionName,
fileId: row.fileId,
...(storageKey === undefined ? {} : { storageKey }),
})
await deps.data.forScope(scope).of(documents).delete(row.id)
}
async function readFileBytes(
deps: KnowledgeDeps,
scope: KnowledgeScope,
fileId: string,
maxBytes: number,
): Promise<Uint8Array> {
const file = await getFile(deps.files, scope, fileId)
const response = await downloadFile(deps.files, scope, fileId)
if (!response.body) throw Object.assign(new Error("source missing"), { permanent: "not_indexable" })
const bytes = await readAll(response.body, maxBytes)
if (!file.sha256 || (await sha256Hex(bytes)) !== file.sha256)
throw Object.assign(new Error("source changed"), { permanent: "file_changed" })
return bytes
}
function decodeText(bytes: Uint8Array): string {
try {
return new TextDecoder("utf-8", { fatal: true }).decode(bytes)
} catch {
throw Object.assign(new Error("the file is not valid UTF-8 text"), { permanent: "not_text" })
}
}
/** Maps an indexing failure to the stable code recorded on the row; null when it is transient. */
function permanentFailure(error: unknown): string | null {
const code = (error as { permanent?: unknown }).permanent
return (INDEX_FAILURES as readonly unknown[]).includes(code) ? (code as IndexFailure) : null
}
+180
View File
@@ -0,0 +1,180 @@
// The ports the knowledge module's use cases are written against, and the tokens the composition
// roots bind. Core imports no SDK and no adapter (spec §2): the search service and the enablement
// platform are shapes, and adapters/ is the only place an SDK exists.
import { type Container, type TenantScope, token } from "@conusai-kit/core"
import type { DataSource } from "@conusai-kit/data"
import type { FilesDeps } from "@conusai-kit/files"
import type { EnablementReason, KnowledgeRefusal, ServiceState } from "../contract/index.js"
import { config, KNOWLEDGE_FLAG_DEFAULTS } from "../contract/index.js"
/** A project scope: collections and documents live at this level. */
export type KnowledgeScope = TenantScope & { workspaceId: string; projectId: string }
/** An org scope: the enablement configuration is the organization's own. */
export type KnowledgeOrgScope = TenantScope & { orgId: string }
export interface KnowledgeDeps {
readonly data: DataSource
readonly files: FilesDeps
readonly flags: KnowledgeFlags
readonly beforeIndex?: (subjectUserId: string) => Promise<void>
/**
* The search service. Present for the effect procedure (search); a read or write procedure's deps
* never carry it, because resolving an effect port from either is a kind violation — the write
* only records the row and publishes the event, and the status read asks `KNOWLEDGE_READINESS`.
*/
readonly search?: KnowledgeSearchService | undefined
/** The document reader and the passage index, for the automatic PDF intake (core-11); bound in the worker. */
readonly reader?: KnowledgeReader | undefined
readonly passages?: KnowledgePassageIndex | undefined
/** The organization's eligibility and indexing quota, which the PDF intake checks before each read. */
readonly enablement?: DocumentAiPlatform | undefined
}
/** One cited search answer. */
export interface KnowledgeHit {
fileId: string
filename: string
collectionName: string
/** 0..1; higher is closer. */
score: number
chunk: number
snippet: string
page?: number
blockId?: string
chunkId?: string
}
/**
* The search service: index a document, search a collection with citations, remove a document, and
* report readiness. Its one adapter is the platform's managed documents-index service (ADR-061), which
* owns chunking, embeddings and the vector store. Every call leaves the process, so this port is an
* effect. `refusal` is why the service will not serve this deployment, when that is the answer.
*/
export interface KnowledgeSearchService {
readiness(scope: KnowledgeScope): Promise<{ state: ServiceState; reasons: string[]; refusal?: KnowledgeRefusal }>
index(
scope: KnowledgeScope,
input: { collectionName: string; fileId: string; filename: string; bytes: Uint8Array; signal?: AbortSignal },
): Promise<void>
search(
scope: KnowledgeScope,
input: { collectionName: string; query: string; limit: number; actorId: string },
): Promise<readonly KnowledgeHit[]>
/** `storageKey` names the object, resolved when the removal was asked for; only an older record lacks it, and then the file is looked up. */
remove(scope: KnowledgeScope, input: { collectionName: string; fileId: string; storageKey?: string }): Promise<void>
}
/**
* One passage of a read PDF, as the automatic intake indexes it: its page (1-based), its place on that
* page (0-based; with the page, its stable name across reads of the same document), the document's own
* block ids it holds lines of, and its text.
*/
export interface KnowledgePassage {
readonly page: number
readonly index: number
readonly blockIds: readonly string[]
readonly text: string
}
/**
* The platform's document reader, as the automatic intake uses it: a PDF in, its passages out. The
* same `idempotencyKey` reads the same document once. A document it can never make passages of throws
* with `permanent: "not_indexable"`; anything else is transient, and the queue retries the job.
*/
export interface KnowledgeReader {
read(
bytes: Uint8Array,
options: { filename: string; idempotencyKey: string; signal?: AbortSignal },
): Promise<readonly KnowledgePassage[]>
}
/**
* The knowledge service's inline index, one item per passage, each under the file's reference plus its
* page and place. The reference is asked for first, so the caller can record what it is about to index
* before any of it is sent, and `remove` takes it back because the file may be gone by then: the service
* removes a reference only exactly, so removal names every passage that was indexed.
*/
export interface KnowledgePassageIndex {
reference(scope: KnowledgeScope, fileId: string): Promise<string>
index(
scope: KnowledgeScope,
input: {
collectionName: string
documentRef: string
fileId: string
passages: readonly KnowledgePassage[]
signal?: AbortSignal
},
): Promise<void>
remove(
scope: KnowledgeScope,
input: { documentRef: string; passages: ReadonlyArray<Pick<KnowledgePassage, "page" | "index">> },
): Promise<void>
}
/**
* What the status read asks of the deployment's search backend. It is a port of its own because a read
* handler may not resolve `KNOWLEDGE_SEARCH`, and must not build it: a factory runs with the resolving
* handler's restrictions, so a search service built for a read would hold a read-only data source.
*/
export type KnowledgeReadiness = Pick<KnowledgeSearchService, "readiness">
/**
* The enablement platform: the checks the enablement flow needs, over the platform's own APIs.
* `checkEligibility` is where the plan capability and the rollout flags surface as typed reasons;
* `indexingQuota` answers today's remaining indexing volume.
*/
export interface DocumentAiPlatform {
checkEligibility(
scope: KnowledgeOrgScope,
): Promise<{ ok: true } | { ok: false; reason: EnablementReason; detail?: string }>
indexingQuota(scope: KnowledgeOrgScope): Promise<number>
}
/** What a flag service answers for the project a call runs in. */
export interface KnowledgeFlags {
enabled(flag: keyof typeof KNOWLEDGE_FLAG_DEFAULTS, scope: KnowledgeScope): boolean | Promise<boolean>
}
/**
* Bound by an app that evaluates flags (the platform's flag service). Unbound: knowledge is on for an
* organization once an administrator has enabled document AI, and off until then (`enablementFlags`).
*/
export const KNOWLEDGE_FLAGS = token<KnowledgeFlags>("knowledge.flags")
/** The organization's own switch: the row `knowledge.settings.enableDocumentAi` writes. */
export const enablementFlags = (data: DataSource): KnowledgeFlags => ({
enabled: async (flag, scope) => {
const row = (await data.forScope(scope).of(config).list({ limit: 1 })).items[0]
return row?.enabled ?? KNOWLEDGE_FLAG_DEFAULTS[flag]
},
})
export const flagsOf = (container: Container, data: DataSource): KnowledgeFlags =>
container.has(KNOWLEDGE_FLAGS) ? container.resolve(KNOWLEDGE_FLAGS) : enablementFlags(data)
/** Bound by the worker and the web shell with the module's search backend adapter. */
export const KNOWLEDGE_SEARCH = token<KnowledgeSearchService>("knowledge.search", { access: "effect" })
/** Bound by the worker with the platform's document reader: the automatic PDF intake reads through it. */
export const KNOWLEDGE_READER = token<KnowledgeReader>("knowledge.reader", { access: "effect" })
/** Bound by the worker with the knowledge service's inline index: the PDF intake's passages, and their removal. */
export const KNOWLEDGE_PASSAGES = token<KnowledgePassageIndex>("knowledge.passages", { access: "effect" })
/** Bound by the web shell with the readiness of the service `KNOWLEDGE_SEARCH` serves. */
export const KNOWLEDGE_READINESS = token<KnowledgeReadiness>("knowledge.readiness")
/**
* Bound by the web shell (the enablement flow runs in a request) and by the worker (the PDF intake checks
* eligibility and quota again before each read, core-11).
*/
export const KNOWLEDGE_ENABLEMENT = token<DocumentAiPlatform>("knowledge.enablement", { access: "effect" })
/** The per-run scope, under the module's own name (decision AT-F6). */
export const KNOWLEDGE_SCOPE = token<TenantScope>("knowledge.scope")
export const knowledgeEnabled = async (deps: Pick<KnowledgeDeps, "flags">, scope: KnowledgeScope): Promise<boolean> =>
deps.flags.enabled("knowledge.enabled", scope)
+65
View File
@@ -0,0 +1,65 @@
// The search: one collection, one query, cited answers. The query leaves the process (the procedure
// is an effect), the ranking is the platform's service, and every answer carries where it came from:
// the file, the chunk, and the snippet a person reads.
import { AppError, forbidden, internal } from "@conusai-kit/core"
import { getFile } from "@conusai-kit/files"
import { documents, MAX_SEARCH_RESULTS, type SearchResult } from "../contract/index.js"
import { requireCollection } from "./collections.js"
import { type KnowledgeDeps, type KnowledgeScope, knowledgeEnabled } from "./ports.js"
export async function searchKnowledge(
deps: KnowledgeDeps,
scope: KnowledgeScope,
input: { collectionName: string; query: string; limit?: number | undefined },
): Promise<{ results: SearchResult[] }> {
if (!(await knowledgeEnabled(deps, scope))) throw forbidden("knowledge is switched off for this project")
await requireCollection(deps, scope, input.collectionName)
const service = deps.search
if (!service) throw internal("the knowledge search service is not bound in this process")
const limit = Math.min(input.limit ?? 10, MAX_SEARCH_RESULTS)
const hits = await service.search(scope, {
collectionName: input.collectionName,
query: input.query,
limit,
actorId: scope.actor.userId,
})
const results: SearchResult[] = []
for (const hit of hits) {
// The current scoped source of truth revokes a citation before the remote deletion job finishes.
const current = (
await deps.data
.forScope(scope)
.of(documents)
.list({
where: [
{ field: "collectionName", op: "eq", value: input.collectionName },
{ field: "fileId", op: "eq", value: hit.fileId },
{ field: "status", op: "eq", value: "indexed" },
],
limit: 1,
})
).items[0]
if (!current || hit.collectionName !== input.collectionName) continue
// The original file can be deleted before its outbox cleanup is delivered.
// Current scoped file visibility is mandatory, independent of stale index rows.
try {
await getFile(deps.files, scope, hit.fileId)
} catch (error) {
if (error instanceof AppError && error.status === 404) continue
throw error
}
results.push({
fileId: hit.fileId,
filename: hit.filename,
collectionName: hit.collectionName,
snippet: hit.snippet,
score: hit.score,
chunk: hit.chunk,
...(hit.page === undefined ? {} : { page: hit.page }),
...(hit.blockId === undefined ? {} : { blockId: hit.blockId }),
...(hit.chunkId === undefined ? {} : { chunkId: hit.chunkId }),
})
}
return { results }
}
+108
View File
@@ -0,0 +1,108 @@
// The enablement flow: one idempotent procedure pair an organization administrator runs, with the
// platform's checks in front of any configuration change and every denial a typed reason code the
// UI maps to an upgrade or policy state.
//
// enableDocumentAi: already enabled -> answers the current configuration, writes nothing.
// Otherwise: eligibility (plan capability + rollout flags) -> quota -> one transaction persists
// the configuration. Nothing is provisioned: retrieval is the platform's service (ADR-061), and an
// app keeps no vector store. A failure at any step writes nothing and is retirable: the same call
// again resumes from the first failing check.
//
// disableDocumentAi: preserves everything — collections, documents, the service's index and the
// recorded configuration stay; only `enabled` flips, so re-enabling resumes without duplication.
import { AppError } from "@conusai-kit/core"
import { config, type EnablementReason, type KnowledgeBackend } from "../contract/index.js"
import type { DocumentAiPlatform, KnowledgeOrgScope } from "./ports.js"
/** A typed enablement denial: the problem's code is the stable reason the UI maps to a state. */
export function enablementDenied(reason: EnablementReason, detail?: string): AppError {
const status = reason === "quota_exhausted" ? 429 : 403
const title =
reason === "plan_required"
? "The organization's plan does not include document AI"
: reason === "feature_disabled"
? "Document AI is disabled by policy"
: "The indexing quota is exhausted"
return new AppError({
type: `/problems/${reason}`,
title,
status,
code: reason,
...(detail === undefined ? {} : { detail }),
})
}
export interface SettingsView {
enabled: boolean
/** Who serves this organization once it has a configuration: only the platform does. */
backend: KnowledgeBackend | null
}
async function currentConfig(deps: { data: import("@conusai-kit/data").DataSource }, scope: KnowledgeOrgScope) {
const row = (await deps.data.forScope(scope).of(config).list({ limit: 1 })).items[0]
return row ?? null
}
export async function getSettings(
deps: { data: import("@conusai-kit/data").DataSource },
scope: KnowledgeOrgScope,
): Promise<SettingsView> {
const row = await currentConfig(deps, scope)
return { enabled: row?.enabled ?? false, backend: row ? "platform" : null }
}
export interface SettingsDeps {
data: import("@conusai-kit/data").DataSource
platform: DocumentAiPlatform
now?: () => number
}
export async function enableDocumentAi(deps: SettingsDeps, scope: KnowledgeOrgScope): Promise<SettingsView> {
const existing = await currentConfig(deps, scope)
if (existing?.enabled) return { enabled: true, backend: "platform" }
const eligibility = await deps.platform.checkEligibility(scope)
if (!eligibility.ok) throw enablementDenied(eligibility.reason, eligibility.detail)
const remaining = await deps.platform.indexingQuota(scope)
if (remaining <= 0) throw enablementDenied("quota_exhausted", "the indexing volume for today is spent")
await deps.data.unitOfWork.run(scope, async (tx) => {
const row = (await tx.of(config).list({ limit: 1 })).items[0]
const patch = {
enabled: true,
backend: "platform" as const,
source: "managed" as const,
serviceId: null,
url: null,
updatedBy: scope.actor.userId,
configuredAt: new Date((deps.now ?? Date.now)()).toISOString(),
}
if (row) await tx.of(config).update(row.id, patch, row.version)
else await tx.of(config).insert(patch)
})
return { enabled: true, backend: "platform" }
}
export async function disableDocumentAi(
deps: Pick<SettingsDeps, "data" | "now">,
scope: KnowledgeOrgScope,
): Promise<{ enabled: false }> {
await deps.data.unitOfWork.run(scope, async (tx) => {
const row = (await tx.of(config).list({ limit: 1 })).items[0]
if (!row?.enabled) return
// Everything else is preserved on purpose: the recorded configuration, every collection and
// document, and the service's index. Disabling fences new work; it deletes nothing.
await tx.of(config).update(
row.id,
{
enabled: false,
updatedBy: scope.actor.userId,
configuredAt: new Date((deps.now ?? Date.now)()).toISOString(),
},
row.version,
)
})
return { enabled: false }
}
+49
View File
@@ -0,0 +1,49 @@
// The status a page reads: the module flag, the search service's readiness (separate from the
// indexing indicator, which is the per-document rows), and the document counts by progress state.
import { type DocumentStatus, documents, type KnowledgeRefusal } from "../contract/index.js"
import { type KnowledgeDeps, type KnowledgeReadiness, type KnowledgeScope, knowledgeEnabled } from "./ports.js"
const PAGE = 200
export interface KnowledgeStatus {
enabled: boolean
service: {
state: "registered" | "indexing" | "ready" | "degraded" | "disabled" | "unavailable"
reasons: string[]
refusal?: KnowledgeRefusal
}
documents: Record<DocumentStatus, number>
}
/** What the status read needs; `readiness` is absent in a process that binds no search backend. */
export type KnowledgeStatusDeps = Pick<KnowledgeDeps, "data" | "flags"> & {
readonly readiness?: KnowledgeReadiness | undefined
}
export async function knowledgeStatusOf(deps: KnowledgeStatusDeps, scope: KnowledgeScope): Promise<KnowledgeStatus> {
const counts: Record<DocumentStatus, number> = { queued: 0, indexing: 0, indexed: 0, failed: 0, cancelled: 0 }
let cursor: string | undefined
for (;;) {
const page = await deps.data
.forScope(scope)
.of(documents)
.list({ limit: PAGE, ...(cursor === undefined ? {} : { cursor }) })
for (const row of page.items) counts[row.status as DocumentStatus] += 1
if (!page.hasMore) break
cursor = page.cursor
}
const enabled = await knowledgeEnabled(deps, scope)
const readiness =
deps.readiness === undefined
? { state: "unavailable" as const, reasons: ["the knowledge search service is not bound in this process"] }
: await deps.readiness.readiness(scope).catch((error: unknown) => ({
state: "unavailable" as const,
reasons: [
error instanceof Error
? (error.message.split("\n")[0] ?? error.message)
: "the search service did not answer",
],
}))
return { enabled, service: readiness, documents: counts }
}
+122
View File
@@ -0,0 +1,122 @@
// English messages of the knowledge module. Every string a page of this module shows is a key here
// and in lt.ts (spec §10.4); `__tests__/i18n.test.ts` fails when the two differ.
export const en = {
title: "Knowledge",
intro:
"Search this project's documents by meaning, on the platform's knowledge service. Files are indexed in the background; answers cite the file and the passage they came from.",
refusals: {
configuration_missing: "Set CONUSAI_ENVIRONMENT for this deployment before using knowledge search.",
plan_required: "The organization's plan does not include document search.",
feature_disabled: "Document search is switched off for this organization by platform policy.",
quota_exhausted: "Today's indexing allowance is spent. Try again tomorrow.",
budget_exhausted: "The organization's AI budget is spent. An administrator can raise it on the platform.",
scope_missing:
"This deployment's token does not hold knowledge.collections:use. Deploy the app again, then have an owner or admin approve it.",
indexing: "The service is building its index. Try again in a few minutes.",
approval_required: "An administrator must approve this on the platform first.",
runtime_unavailable: "The knowledge service did not answer. Try again later.",
},
failures: {
processing_exhausted: "Processing stopped after its final retry. Check services and permissions before retrying.",
file_changed: "The source file changed. Upload the original again.",
not_text: "The file is not text.",
not_indexable: "This kind of file cannot be indexed.",
too_large: "The file is too large to index.",
},
status: {
heading: "Service status",
flagOff:
"Knowledge is switched off. An organization administrator enables document AI in the organization's settings.",
state: "Search service",
stateReady: "Ready",
stateIndexing: "Indexing",
stateRegistered: "Registered",
stateDegraded: "Degraded",
stateDisabled: "Disabled by policy",
stateUnavailable: "Unavailable",
reasons: "Reasons",
indexing: "Indexing in progress",
documents: "{queued} queued, {indexing} indexing, {indexed} indexed, {failed} failed",
},
collections: {
heading: "Collections",
empty: "No collections yet.",
name: "Name",
documents: "Documents",
indexed: "Indexed",
createHeading: "New collection",
nameLabel: "Name",
nameHint: "Lower-case letters, digits and dashes.",
create: "Create collection",
creating: "Creating",
pickerLabel: "Collection",
pick: "Choose a collection",
},
autoIndex: {
heading: "Index uploaded PDFs automatically",
hint: "Every PDF uploaded to this project is read and indexed into the chosen collection, with page references. Your permission to manage knowledge is checked again for each file.",
label: "Collection",
off: "Off",
submit: "Save",
busy: "Saving…",
done: "Saved.",
},
upload: {
heading: "Index a document",
label: "File",
hint: "Plain text, Markdown, JSON or CSV, up to 1 MB.",
submit: "Index",
busy: "Indexing requested",
done: "The document is queued for indexing.",
failed: "That did not work: {detail}",
},
documents: {
heading: "Documents",
empty: "Nothing is indexed in this collection yet.",
file: "File",
status: "Status",
statusQueued: "Queued",
statusIndexing: "Indexing",
statusIndexed: "Indexed",
statusFailed: "Failed",
statusCancelled: "Removed",
error: "Error",
remove: "Remove {filename}",
removeShort: "Remove",
},
search: {
heading: "Search",
label: "Question",
placeholder: "Ask across the indexed documents",
submit: "Search",
busy: "Searching",
empty: "No answers in this collection.",
results: "{count, plural, one {# answer} other {# answers}}",
snippet: "Passage",
source: "Source",
chunk: "result {chunk}",
page: "page {page}",
score: "relevance {score}",
disclosure: "AI-assisted retrieval: answers are ranked by a model and must be checked against the source.",
},
settings: {
heading: "Document AI",
state: "State",
stateOn: "Enabled",
stateOff: "Disabled",
backend: "Backend",
backendPlatform: "The platform's knowledge service",
enable: "Enable document AI",
enabling: "Enabling",
disable: "Disable",
disabling: "Disabling",
enabledDone: "Document AI is enabled.",
disabledDone: "Document AI is disabled. Collections, documents and the index are preserved.",
planRequired: "The organization's plan does not include document AI.",
featureDisabled: "Document AI is disabled by platform policy.",
quotaExhausted: "The indexing quota is exhausted; try again later.",
upgrade: "Upgrade the plan to enable document AI.",
hint: "Enabling checks the plan and the rollout flags and records the choice in this organization; the platform's knowledge service keeps the index. Disabling keeps every collection, document and indexed passage.",
},
} as const
+2
View File
@@ -0,0 +1,2 @@
export { en } from "./en.js"
export { lt } from "./lt.js"
+126
View File
@@ -0,0 +1,126 @@
// Lietuviški knowledge modulio pranešimai. Kiekvienas puslapio tekstas yra raktas čia ir en.ts
// faile; `__tests__/i18n.test.ts` žlunga, kai jie skiriasi.
import type { en } from "./en.js"
type Messages<T> = { [K in keyof T]: T[K] extends string ? string : Messages<T[K]> }
export const lt: Messages<typeof en> = {
title: "Žinios",
intro:
"Ieškokite šio projekto dokumentų pagal prasmę platformos žinių paslaugoje. Failai indeksuojami fone; atsakymai nurodo failą ir ištrauką, iš kurios paimti.",
refusals: {
configuration_missing: "Prieš naudodami žinių paiešką šiame diegime nustatykite CONUSAI_ENVIRONMENT.",
plan_required: "Organizacijos planas dokumentų paieškos neapima.",
feature_disabled: "Dokumentų paiešką šiai organizacijai išjungė platformos politika.",
quota_exhausted: "Šios dienos indeksavimo limitas išnaudotas. Bandykite rytoj.",
budget_exhausted: "Organizacijos DI biudžetas išnaudotas. Administratorius gali jį padidinti platformoje.",
scope_missing:
"Šio diegimo prieigos raktas neturi leidimo knowledge.collections:use. Įdiekite programą iš naujo, tada paprašykite savininko ar administratoriaus jį patvirtinti.",
indexing: "Paslauga kuria indeksą. Bandykite po kelių minučių.",
approval_required: "Pirmiausia tai platformoje turi patvirtinti administratorius.",
runtime_unavailable: "Žinių paslauga neatsakė. Bandykite vėliau.",
},
failures: {
processing_exhausted:
"Apdorojimas sustabdytas po paskutinio bandymo. Prieš kartodami patikrinkite paslaugas ir teises.",
file_changed: "Pasikeitė šaltinio failas. Dar kartą įkelkite originalą.",
not_text: "Failas nėra tekstinis.",
not_indexable: "Tokio tipo failo indeksuoti negalima.",
too_large: "Failas per didelis, kad būtų suindeksuotas.",
},
status: {
heading: "Paslaugos būsena",
flagOff: "Žinios išjungtos. Dokumentų DI organizacijos nustatymuose įjungia organizacijos administratorius.",
state: "Paieškos paslauga",
stateReady: "Pasiruošusi",
stateIndexing: "Indeksuoja",
stateRegistered: "Registruota",
stateDegraded: "Nukentėjusi",
stateDisabled: "Išjungta politikos",
stateUnavailable: "Neprieinama",
reasons: "Priežastys",
indexing: "Vyksta indeksavimas",
documents: "{queued} laukia, {indexing} indeksuojama, {indexed} suindeksuota, {failed} nepavyko",
},
collections: {
heading: "Kolekcijos",
empty: "Kolekcijų dar nėra.",
name: "Pavadinimas",
documents: "Dokumentai",
indexed: "Suindeksuota",
createHeading: "Nauja kolekcija",
nameLabel: "Pavadinimas",
nameHint: "Mažosios raidės, skaičiai ir brūkšneliai.",
create: "Sukurti kolekciją",
creating: "Kuriama",
pickerLabel: "Kolekcija",
pick: "Pasirinkite kolekciją",
},
autoIndex: {
heading: "Automatiškai indeksuoti įkeltus PDF",
hint: "Kiekvienas į šį projektą įkeltas PDF perskaitomas ir indeksuojamas pasirinktoje kolekcijoje su puslapių nuorodomis. Jūsų teisė tvarkyti žinias tikrinama iš naujo kiekvienam failui.",
label: "Kolekcija",
off: "Išjungta",
submit: "Išsaugoti",
busy: "Išsaugoma…",
done: "Išsaugota.",
},
upload: {
heading: "Indeksuoti dokumentą",
label: "Failas",
hint: "Grynasis tekstas, Markdown, JSON arba CSV, iki 1 MB.",
submit: "Indeksuoti",
busy: "Indeksavimas paprašytas",
done: "Dokumentas eilėje indeksavimui.",
failed: "Nepavyko: {detail}",
},
documents: {
heading: "Dokumentai",
empty: "Šioje kolekcijoje dar nieko suindeksuota.",
file: "Failas",
status: "Būsena",
statusQueued: "Laukia",
statusIndexing: "Indeksuojama",
statusIndexed: "Suindeksuota",
statusFailed: "Nepavyko",
statusCancelled: "Pašalinta",
error: "Klaida",
remove: "Pašalinti: {filename}",
removeShort: "Pašalinti",
},
search: {
heading: "Paieška",
label: "Klausimas",
placeholder: "Klauskite per suindeksuotus dokumentus",
submit: "Ieškoti",
busy: "Ieškoma",
empty: "Šioje kolekcijoje atsakymų nėra.",
results: "{count, plural, one {# atsakymas} few {# atsakymai} many {# atsakymo} other {# atsakymų}}",
snippet: "Ištrauka",
source: "Šaltinis",
chunk: "{chunk} rezultatas",
page: "{page} puslapis",
score: "atitiktis {score}",
disclosure: "Paieška su DI pagalba: atsakymus rikiuoja modelis, juos būtina patikrinti pagal šaltinį.",
},
settings: {
heading: "Dokumentų DI",
state: "Būsena",
stateOn: "Įjungta",
stateOff: "Išjungta",
backend: "Backend'as",
backendPlatform: "Platformos žinių paslauga",
enable: "Įjungti dokumentų DI",
enabling: "Įjungiama",
disable: "Išjungti",
disabling: "Išjungiama",
enabledDone: "Dokumentų DI įjungta.",
disabledDone: "Dokumentų DI išjungta. Kolekcijos, dokumentai ir indeksas išsaugoti.",
planRequired: "Organizacijos planas dokumentų DI neapima.",
featureDisabled: "Dokumentų DI išjungta platformos politikos.",
quotaExhausted: "Indeksavimo kvota išsemta; bandykite vėliau.",
upgrade: "Atnaujinkite planą, kad įjungtumėte dokumentų DI.",
hint: "Įjungdama tikrina planą ir išleidimo požymius ir įrašo pasirinkimą šioje organizacijoje; indeksą saugo platformos žinių paslauga. Išjungdama viską išsaugo.",
},
}
+34
View File
@@ -0,0 +1,34 @@
// The knowledge module's public surface: the manifest and what the composition roots bind.
export type { SdkDocumentAiPlatformOptions } from "./adapters/enablement.js"
export { AGENT_DOCUMENT_TOOLS_FLAG, AGENT_KNOWLEDGE_TOOLS_FLAG, sdkDocumentAiPlatform } from "./adapters/enablement.js"
export { platformKnowledgeSearch, platformPassageIndex } from "./adapters/platform-knowledge.js"
export type { SdkKnowledgeReaderOptions } from "./adapters/reader.js"
export { sdkKnowledgeReader } from "./adapters/reader.js"
export { platformKnowledgeReadiness } from "./adapters/readiness.js"
export * from "./contract/index.js"
export type {
DocumentAiPlatform,
KnowledgeDeps,
KnowledgeFlags,
KnowledgeHit,
KnowledgeOrgScope,
KnowledgePassage,
KnowledgePassageIndex,
KnowledgeReader,
KnowledgeReadiness,
KnowledgeScope,
KnowledgeSearchService,
} from "./core/ports.js"
export {
KNOWLEDGE_ENABLEMENT,
KNOWLEDGE_FLAGS,
KNOWLEDGE_PASSAGES,
KNOWLEDGE_READER,
KNOWLEDGE_READINESS,
KNOWLEDGE_SCOPE,
KNOWLEDGE_SEARCH,
} from "./core/ports.js"
export { convexKnowledgeModule } from "./module.convex.js"
export { knowledgeModule } from "./module.js"
export { KNOWLEDGE_MODULE_BASE } from "./module.shared.js"
@@ -0,0 +1,68 @@
-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0
create table "knowledge_collection" (
"id" uuid primary key,
"version" integer not null,
"created_at" timestamptz not null,
"updated_at" timestamptz not null,
"org_id" uuid not null,
"workspace_id" uuid not null,
"project_id" uuid not null,
"name" text not null,
"created_by" text not null
);
alter table "knowledge_collection" add constraint "knowledge_collection_org_id_workspace_id_project_id_name_key" unique nulls not distinct ("org_id", "workspace_id", "project_id", "name");
create table "knowledge_config" (
"id" uuid primary key,
"version" integer not null,
"created_at" timestamptz not null,
"updated_at" timestamptz not null,
"org_id" uuid not null,
"enabled" boolean not null,
"backend" text not null,
"source" text not null,
"service_id" text,
"url" text,
"updated_by" text not null,
"configured_at" text not null
);
create table "knowledge_document" (
"id" uuid primary key,
"version" integer not null,
"created_at" timestamptz not null,
"updated_at" timestamptz not null,
"org_id" uuid not null,
"workspace_id" uuid not null,
"project_id" uuid not null,
"collection_name" text not null,
"file_id" uuid not null,
"filename" text not null,
"status" text not null,
"error" text,
"indexed_at" text,
"requested_by" text not null
);
alter table "knowledge_document" add constraint "knowledge_document_org_id_workspace_id_project_id_12648b24_key" unique nulls not distinct ("org_id", "workspace_id", "project_id", "collection_name", "file_id");
create index "knowledge_document_org_id_workspace_id_project_id_ec03a581_idx" on "knowledge_document" ("org_id", "workspace_id", "project_id", "collection_name");
alter table "knowledge_collection" enable row level security;
alter table "knowledge_collection" force row level security;
create policy "knowledge_collection_tenant" on "knowledge_collection" using ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid) with check ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid);
alter table "knowledge_config" enable row level security;
alter table "knowledge_config" force row level security;
create policy "knowledge_config_tenant" on "knowledge_config" using ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid) with check ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid);
alter table "knowledge_document" enable row level security;
alter table "knowledge_document" force row level security;
create policy "knowledge_document_tenant" on "knowledge_document" using ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid) with check ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid);
@@ -0,0 +1,19 @@
-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0
create table "knowledge_auto_index" (
"id" uuid primary key,
"version" integer not null,
"created_at" timestamptz not null,
"updated_at" timestamptz not null,
"org_id" uuid not null,
"workspace_id" uuid not null,
"project_id" uuid not null,
"collection_name" text not null,
"enabled_by" text not null,
"enabled_at" text not null
);
alter table "knowledge_auto_index" enable row level security;
alter table "knowledge_auto_index" force row level security;
create policy "knowledge_auto_index_tenant" on "knowledge_auto_index" using ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid) with check ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid);
@@ -0,0 +1,22 @@
-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0
create table "knowledge_passage_set" (
"id" uuid primary key,
"version" integer not null,
"created_at" timestamptz not null,
"updated_at" timestamptz not null,
"org_id" uuid not null,
"workspace_id" uuid not null,
"project_id" uuid not null,
"collection_name" text not null,
"file_id" uuid not null,
"document_ref" text not null,
"pages" text not null
);
alter table "knowledge_passage_set" add constraint "knowledge_passage_set_org_id_workspace_id_project_id_collection_name_file_id_key" unique nulls not distinct ("org_id", "workspace_id", "project_id", "collection_name", "file_id");
alter table "knowledge_passage_set" enable row level security;
alter table "knowledge_passage_set" force row level security;
create policy "knowledge_passage_set_tenant" on "knowledge_passage_set" using ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid) with check ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid);
+24
View File
@@ -0,0 +1,24 @@
// The Convex profile binding of the knowledge module, shipped fail-closed for a future wiring:
// the integration review kept the module Postgres-only because the search effect, the
// index/remove jobs and the enablement flow all leave the process and have no composed Convex
// Node actions yet (see README.md). Knowledge indexing needs the Node runtime boundary (the platform's
// knowledge client), exactly like documents' OCR: the job handlers here fail
// closed so the work can only reach the platform SDKs through a composed Node action, never the
// transaction dispatcher.
import { defineModule, internal, type JobDef } from "@conusai-kit/core"
import { type DocumentJobPayload, documentPayload } from "./adapters/jobs.js"
import { INDEX_JOB, REMOVE_JOB } from "./contract/index.js"
import { KNOWLEDGE_MODULE_BASE } from "./module.shared.js"
const actionOnly = (name: string): JobDef<DocumentJobPayload> => ({
payload: documentPayload,
async handler() {
throw internal(`${name} must run through the Convex Node action boundary`)
},
})
export const convexKnowledgeModule = defineModule({
...KNOWLEDGE_MODULE_BASE,
jobs: { [INDEX_JOB]: actionOnly(INDEX_JOB), [REMOVE_JOB]: actionOnly(REMOVE_JOB) },
})
+98
View File
@@ -0,0 +1,98 @@
// Shared, runtime-neutral knowledge manifest data. The normal app binds its worker implementations
// of the index/remove jobs; the Convex binding file (module.convex.ts) ships fail-closed for a
// future wiring — the module stays Postgres-only until the search effect and the jobs have their
// composed Node actions (see README.md).
import { contribute, type ModuleManifest } from "@conusai-kit/core"
import { repositoryBindings } from "@conusai-kit/data"
import { filesProcessors } from "@conusai-kit/files"
import { accountData } from "@conusai-kit/tenancy"
import { knowledgeAccountData } from "./adapters/account.js"
import { KNOWLEDGE_PROCEDURE_IMPLS } from "./adapters/procedures.js"
import {
AUTO_INDEX_CHANGED,
AUTO_INDEX_CONTENT_TYPES,
AUTO_INDEX_JOB,
AUTO_INDEX_PROCESSOR_KEY,
COLLECTION_CREATED,
DOCUMENT_AI_DISABLED,
DOCUMENT_AI_ENABLED,
INDEX_JOB,
KNOWLEDGE_COLLECTIONS,
KNOWLEDGE_ENTITIES,
KNOWLEDGE_FLAG_DEFAULTS,
} from "./contract/index.js"
import { en, lt } from "./i18n/index.js"
const tables = repositoryBindings(...KNOWLEDGE_ENTITIES)
export const KNOWLEDGE_MODULE_BASE: ModuleManifest = {
id: "knowledge",
version: "0.1.0",
kit: "^0.1.0",
requires: ["tenancy", "files"],
serviceRequirements: [
{
transport: "platform",
service: "knowledge",
apiRange: "=1.0.0",
profiles: ["postgres"],
operations: {
[INDEX_JOB]: ["knowledge.collections:use"],
[AUTO_INDEX_JOB]: ["knowledge.collections:use"],
"knowledge.search": ["knowledge.collections:use"],
},
configuration: ["ai_processing"],
dependencies: ["blob", "ai"],
dataClasses: ["document", "personal", "ai-input"],
partitionRequired: false,
remediation: "/settings/services/knowledge",
},
{
// The automatic PDF intake reads an uploaded PDF through the platform's document reader (core-11).
transport: "platform",
service: "ocr",
apiRange: "=1.0.0-preview.20261103",
profiles: ["postgres"],
operations: { [AUTO_INDEX_JOB]: ["ocr:use"] },
configuration: ["ocr_processing"],
dependencies: ["blob"],
dataClasses: ["document", "personal"],
partitionRequired: false,
remediation: "/settings/services/ocr",
},
],
permissions: {
// Searching and reading the status is a member's act; the kit's member maps onto "editor".
"knowledge.knowledge.search": "editor",
// Managing collections, indexing and removing documents, and the enablement flow are
// administrative acts.
"knowledge.collection.manage": "admin",
"knowledge.knowledge.manage": "admin",
"knowledge.settings.manage": "admin",
},
procedures: KNOWLEDGE_PROCEDURE_IMPLS,
// Indexing is live (TS6.1): a change of knowledge_document reaches every member's open page.
collections: [...KNOWLEDGE_COLLECTIONS],
repositories: { postgres: tables },
migrations: { postgres: "migrations" },
// The module ships the Postgres profile; the Convex binding file (module.convex.ts) ships
// fail-closed for a future wiring and is deliberately NOT registered in convex/ — the search
// effect, the jobs and the enablement flow all leave the process and have no composed Convex
// Node actions yet (see README.md).
profiles: ["postgres"],
audit: [COLLECTION_CREATED, DOCUMENT_AI_ENABLED, DOCUMENT_AI_DISABLED, AUTO_INDEX_CHANGED],
flags: Object.fromEntries(Object.entries(KNOWLEDGE_FLAG_DEFAULTS).map(([flag, value]) => [flag, { default: value }])),
ui: {
nav: [{ scope: "project", label: "knowledge.title", href: "knowledge", permission: "knowledge.knowledge.search" }],
},
i18n: { en, lt },
// An opted-in project's uploaded PDFs, indexed automatically (core-11): the job reads nothing for a
// project that has not opted in.
extends: [
contribute(accountData, [knowledgeAccountData]),
contribute(filesProcessors, [
{ key: AUTO_INDEX_PROCESSOR_KEY, contentTypes: [...AUTO_INDEX_CONTENT_TYPES], job: AUTO_INDEX_JOB },
]),
],
}
+15
View File
@@ -0,0 +1,15 @@
// The knowledge module's manifest (spec §4.1): the one place it registers anything. The normal
// (Postgres) app binds the worker implementations of the index/remove jobs; the module's own
// outbox subscribers enqueue them, so a web request writes the row and answers, never waiting on
// the platform's knowledge service.
import { defineModule } from "@conusai-kit/core"
import { KNOWLEDGE_JOBS, KNOWLEDGE_SUBSCRIPTIONS } from "./adapters/jobs.js"
import { KNOWLEDGE_EVENTS } from "./contract/index.js"
import { KNOWLEDGE_MODULE_BASE } from "./module.shared.js"
export const knowledgeModule = defineModule({
...KNOWLEDGE_MODULE_BASE,
jobs: KNOWLEDGE_JOBS,
events: { publishes: [...KNOWLEDGE_EVENTS], subscribes: KNOWLEDGE_SUBSCRIPTIONS },
})
+66
View File
@@ -0,0 +1,66 @@
"use client"
// The project's automatic PDF indexing (core-11): an administrator picks the collection uploaded PDFs
// go into, or turns it off. The write answers the new state, so the form shows it without re-running
// the page's loader (the module declares live collections; test/live-writes.test.ts).
import type { ProjectSlugs } from "@conusai-kit/ui"
import { useWrite } from "@conusai-kit/ui/data"
import { FormLayout, PageSection, SelectField } from "@conusai-kit/ui/patterns"
import { type ReactElement, useState } from "react"
export interface AutoIndexFormLabels {
readonly heading: string
readonly hint: string
readonly label: string
readonly off: string
readonly submit: string
readonly busy: string
readonly done: string
}
/** The select's value for "off": a collection name is lower-case letters, digits and dashes, never this. */
const OFF = "_off"
export function AutoIndexForm({
slugs,
collections,
current,
labels,
}: {
slugs: ProjectSlugs
collections: readonly string[]
current: string | null
labels: AutoIndexFormLabels
}): ReactElement {
const setAutoIndex = useWrite<ProjectSlugs & { collectionName: string | null }, { collectionName: string | null }>(
"knowledge.autoindex.set",
{ org: slugs.org },
)
const [selected, setSelected] = useState<string | null>(current)
const [done, setDone] = useState(false)
const save = async (_: unknown, data: FormData): Promise<void> => {
const value = data.get("collection")
const collectionName = typeof value === "string" && value !== OFF ? value : null
setDone(false)
const answer = await setAutoIndex.mutate({ ...slugs, collectionName })
setSelected(answer.collectionName)
setDone(true)
}
return (
<PageSection title={labels.heading} level={3} variant="card">
<p>{labels.hint}</p>
<FormLayout submitLabel={labels.submit} busyLabel={labels.busy} onSubmit={save}>
<SelectField
name="collection"
label={labels.label}
defaultValue={selected ?? OFF}
options={[{ value: OFF, label: labels.off }, ...collections.map((name) => ({ value: name, label: name }))]}
/>
</FormLayout>
{done ? <p role="status">{labels.done}</p> : null}
</PageSection>
)
}
+63
View File
@@ -0,0 +1,63 @@
"use client"
// Upload and index: the browser sends the file to the files module (the kit's `uploadFile`), then
// asks this module to index it (`knowledge.document.index`). The indexing itself is a worker job;
// this form only ever waits on the request, which resolves once the live stream has seen the document's
// row, so the collection's live list already shows it queued.
import type { ProjectSlugs } from "@conusai-kit/ui"
import { useUpload, useWrite } from "@conusai-kit/ui/data"
import { FormLayout, PageSection, TextField } from "@conusai-kit/ui/patterns"
import { type ReactElement, useState } from "react"
import { INDEXABLE_CONTENT_TYPES, MAX_INDEX_BYTES } from "../contract/index.js"
export interface IndexFormLabels {
readonly heading: string
readonly label: string
readonly hint: string
readonly submit: string
readonly busy: string
readonly done: string
}
export function IndexForm({
slugs,
collectionName,
labels,
}: {
slugs: ProjectSlugs
collectionName: string
labels: IndexFormLabels
}): ReactElement {
const upload = useUpload()
const indexDocument = useWrite<ProjectSlugs & { collectionName: string; fileId: string }, unknown>(
"knowledge.document.index",
{ org: slugs.org },
)
const [done, setDone] = useState(false)
const index = async (_: unknown, data: FormData): Promise<void> => {
const file = data.get("file")
if (!(file instanceof File) || file.size === 0) return
setDone(false)
const stored = await upload(slugs, file)
await indexDocument.mutate({ ...slugs, collectionName, fileId: stored.fileId })
setDone(true)
}
return (
<PageSection title={labels.heading} level={3} variant="card">
<FormLayout submitLabel={labels.submit} busyLabel={labels.busy} onSubmit={index}>
<TextField
name="file"
type="file"
label={labels.label}
description={`${labels.hint} · ${Math.round(MAX_INDEX_BYTES / 1024)} KB.`}
accept={INDEXABLE_CONTENT_TYPES.join(",")}
required
/>
</FormLayout>
{done ? <p role="status">{labels.done}</p> : null}
</PageSection>
)
}
+8
View File
@@ -0,0 +1,8 @@
export type { IndexFormLabels } from "./index-form.js"
export { IndexForm } from "./index-form.js"
export type { SearchFormLabels, StatusLabels } from "./labels.js"
export { CollectionDocuments, type DocumentListLabels, useLiveDocuments } from "./live-documents.js"
export { knowledgePage } from "./pages.js"
export { SearchForm } from "./search-form.js"
export type { SettingsSectionLabels } from "./settings-section.js"
export { SettingsSection } from "./settings-section.js"
+118
View File
@@ -0,0 +1,118 @@
// The labels each component takes, built from a message lookup. Every key this module's pages use
// is named here and nowhere else, so `__tests__/i18n.test.ts` can run these builders against the
// catalogues and fail on a key either language lacks.
import { KNOWLEDGE_REFUSALS, type KnowledgeRefusal } from "../contract/index.js"
export type Translate = (key: string, values?: Record<string, string | number>) => string
/** A typed refusal as a page or a form shows it. */
export interface RefusalNotice {
readonly code: KnowledgeRefusal
readonly text: string
}
export type RefusalLabels = Readonly<Record<KnowledgeRefusal, string>>
export const refusalLabels = (t: Translate): RefusalLabels =>
Object.fromEntries(KNOWLEDGE_REFUSALS.map((code) => [code, t(`refusals.${code}`)])) as RefusalLabels
/** The typed refusal a procedure answered, or undefined for any other error (which the form words itself). */
export function refusalOf(labels: RefusalLabels, error: unknown): RefusalNotice | undefined {
const code = (error as { problem?: { code?: unknown } } | null)?.problem?.code
return typeof code === "string" && code in labels
? { code: code as KnowledgeRefusal, text: labels[code as KnowledgeRefusal] }
: undefined
}
/** The words for a failure a document row records: the file's own, or the service's refusal; else the code. */
export function failureText(t: Translate, code: string): string {
return failureLabels(t)[code] ?? code
}
/** The failures a document row can name with words of their own: the file's, and the service's refusals. */
const FILE_FAILURES = ["file_changed", "not_text", "not_indexable", "too_large"] as const
/** The words of each failure `failureText` knows, by code, for a page that words a live row in the browser. */
export const failureLabels = (t: Translate): Readonly<Record<string, string>> =>
Object.fromEntries([
...KNOWLEDGE_REFUSALS.map((code) => [code, t(`refusals.${code}`)]),
...FILE_FAILURES.map((code) => [code, t(`failures.${code}`)]),
])
/** The status shape the labels render, declared here: ui reads no core/. */
interface StatusCounts {
documents: { queued: number; indexing: number; indexed: number; failed: number; cancelled: number }
}
export interface StatusLabels {
readonly heading: string
readonly flagOff: string
readonly state: string
readonly reasons: string
readonly indexing: string
readonly documents: string
}
export function statusLabels(t: Translate, status: StatusCounts): StatusLabels {
return {
heading: t("status.heading"),
flagOff: t("status.flagOff"),
state: t("status.state"),
reasons: t("status.reasons"),
indexing: t("status.indexing"),
documents: t("status.documents", status.documents),
}
}
export const serviceStateLabel = (t: Translate, state: string): string => {
switch (state) {
case "ready":
return t("status.stateReady")
case "indexing":
return t("status.stateIndexing")
case "registered":
return t("status.stateRegistered")
case "degraded":
return t("status.stateDegraded")
case "disabled":
return t("status.stateDisabled")
default:
return t("status.stateUnavailable")
}
}
export interface SearchFormLabels {
readonly heading: string
readonly label: string
readonly placeholder: string
readonly submit: string
readonly busy: string
readonly empty: string
readonly results: string
readonly snippet: string
readonly source: string
readonly chunk: string
readonly page: string
readonly disclosure: string
readonly refusals: RefusalLabels
}
export function searchFormLabels(t: Translate): SearchFormLabels {
return {
heading: t("search.heading"),
label: t("search.label"),
placeholder: t("search.placeholder"),
submit: t("search.submit"),
busy: t("search.busy"),
empty: t("search.empty"),
// Not filled in here: only the browser knows the count, and `formatMessage` brings the plural rules.
results: t("search.results"),
snippet: t("search.snippet"),
source: t("search.source"),
chunk: t("search.chunk", { chunk: "{chunk}" }),
page: t("search.page", { page: "{page}" }),
disclosure: t("search.disclosure"),
refusals: refusalLabels(t),
}
}
+96
View File
@@ -0,0 +1,96 @@
"use client"
// One collection's documents, live (umbrella TS6.4 and TS6.6): the page's view renders them and hands them
// over as the seed, so the page arrives complete; this keeps them current through the organization's live
// stream -- a document another member asks to index appears, and each moves from queued to indexed or
// failed as the worker and the platform answer, without a reload. The rows come from
// `knowledge.document.getMany`, under the same permission as the list; a topic is the whole project, so the
// rows of the project's other collections are filtered out here.
import type { ProjectSlugs } from "@conusai-kit/ui"
import { eq, liveCollection, useDataClient, useLive } from "@conusai-kit/ui/data"
import { type ResourceColumn, ResourceList } from "@conusai-kit/ui/patterns"
import { type ReactElement, useState } from "react"
import { DOCUMENT_COLLECTION, type DocumentView } from "../contract/index.js"
export interface DocumentListLabels {
readonly caption: string
readonly file: string
readonly status: string
readonly error: string
/** The message of each document status, by status. */
readonly statuses: Readonly<Record<DocumentView["status"], string>>
/** The words of each failure code a row can carry; an unknown code is shown as it is. */
readonly failures: Readonly<Record<string, string>>
}
export function useLiveDocuments(
slugs: ProjectSlugs,
collectionName: string,
seed: readonly DocumentView[],
): readonly DocumentView[] {
const client = useDataClient()
// One collection per knowledge collection for the life of the page; the seed is the first render's.
const [documents] = useState(() =>
liveCollection<DocumentView>({
client,
org: slugs.org,
topic: { ws: slugs.workspace, project: slugs.project },
entity: DOCUMENT_COLLECTION.entity,
key: DOCUMENT_COLLECTION.key,
id: `${DOCUMENT_COLLECTION.entity}:${slugs.org}:${slugs.workspace}:${slugs.project}:${collectionName}`,
seed,
list: {
procedure: DOCUMENT_COLLECTION.list,
input: { ...slugs, collectionName },
rows: (out) => (out as { documents: DocumentView[] }).documents,
},
getMany: {
procedure: DOCUMENT_COLLECTION.getMany,
input: { ...slugs },
rows: (out) => (out as { items: DocumentView[] }).items,
},
}),
)
const { data, isReady } = useLive(documents, (q, rows) =>
q
.from({ document: rows })
.where(({ document }: { document: DocumentView }) => eq(document.collectionName, collectionName)),
)
// Until the collection is ready the server's rows are the answer, so the first paint matches it.
return isReady ? data : seed
}
export function CollectionDocuments({
slugs,
collectionName,
seed,
labels,
}: {
slugs: ProjectSlugs
collectionName: string
seed: readonly DocumentView[]
labels: DocumentListLabels
}): ReactElement {
const rows = useLiveDocuments(slugs, collectionName, seed)
const columns: ResourceColumn<DocumentView>[] = [
{ id: "filename", header: labels.file, wrap: true },
{ id: "status", header: labels.status, cell: (document) => labels.statuses[document.status] },
{
id: "error",
header: labels.error,
wrap: true,
cell: (document) => (document.error === null ? "" : (labels.failures[document.error] ?? document.error)),
},
]
return (
<ResourceList
label={labels.caption}
columns={columns}
data={rows}
rowId={(document) => document.documentId}
rowLabel={(document) => document.filename}
empty={{ title: labels.caption }}
/>
)
}
+207
View File
@@ -0,0 +1,207 @@
// The knowledge module's page, on the shell's F10 contract (docs/reference/app-kit.md): a loader the web
// shell runs on the server under `/<org>/<workspace>/<project>/knowledge`, and the view of what it
// returned. Every read is a procedure through the same caller the API uses, and every label is translated
// in the loader; every write is a client component that calls a procedure through the shell's transport.
//
// The page renders three things separately on purpose: the service's health (the module flag and
// the search service's readiness), the indexing indicator (the per-document progress states), and
// the search itself — "indexing" is not "down", and the page never conflates them.
import { type Container, notFound, type TransportCaller } from "@conusai-kit/core"
import type { I18n } from "@conusai-kit/i18n"
import { type ProjectPage, type ProjectPageViewProps, projectPage } from "@conusai-kit/ui"
import { FactList, Page, PageHeader, PageSection } from "@conusai-kit/ui/patterns"
import { Alert, AlertDescription } from "@conusai-kit/ui/primitives"
import type { ReactElement } from "react"
import {
autoIndexGet,
type CollectionView,
collectionList,
type DocumentView,
documentList,
knowledgeStatus,
} from "../contract/index.js"
import { AutoIndexForm, type AutoIndexFormLabels } from "./auto-index-form.js"
import { IndexForm, type IndexFormLabels } from "./index-form.js"
import {
failureLabels,
type RefusalNotice,
refusalLabels,
type SearchFormLabels,
type StatusLabels,
searchFormLabels,
serviceStateLabel,
statusLabels,
} from "./labels.js"
import { CollectionDocuments, type DocumentListLabels } from "./live-documents.js"
import { SearchForm } from "./search-form.js"
/** One collection and its documents: the seed of its live list. */
export interface KnowledgeCollectionRow {
readonly name: string
readonly documents: readonly DocumentView[]
}
export interface KnowledgePageData {
readonly title: string
readonly intro: string
readonly status: {
readonly enabled: boolean
readonly labels: StatusLabels
/** The service's state in the reader's language. */
readonly state: string
readonly reasons: readonly string[]
/** Why the service will not serve this deployment, when that is the answer. */
readonly refusal?: RefusalNotice
}
readonly collections: readonly KnowledgeCollectionRow[]
readonly documents: DocumentListLabels
readonly index: IndexFormLabels
readonly search: SearchFormLabels
/** The project's automatic PDF indexing: the collection it fills, or null when it is off. */
readonly autoIndex: { readonly current: string | null; readonly labels: AutoIndexFormLabels }
}
export type KnowledgePage = ProjectPage<TransportCaller, I18n, Container, KnowledgePageData>
const indexFormLabels = (t: (key: string, values?: Record<string, string | number>) => string): IndexFormLabels => ({
heading: t("upload.heading"),
label: t("upload.label"),
hint: t("upload.hint"),
submit: t("upload.submit"),
busy: t("upload.busy"),
done: t("upload.done"),
})
const documentStatusKey: Record<DocumentView["status"], string> = {
queued: "documents.statusQueued",
indexing: "documents.statusIndexing",
indexed: "documents.statusIndexed",
failed: "documents.statusFailed",
cancelled: "documents.statusCancelled",
}
export const knowledgePage: KnowledgePage = projectPage({
async load({ slugs, segments, caller, i18n }): Promise<KnowledgePageData> {
const t = (key: string, values?: Record<string, string | number>) => i18n.message(`knowledge.${key}`, values)
if (segments.length > 0) throw notFound("no such page")
const [{ collections }, status, auto] = await Promise.all([
caller.call(collectionList, slugs),
caller.call(knowledgeStatus, slugs),
caller.call(autoIndexGet, slugs),
])
const rows = await Promise.all(
collections.map(async (collection: CollectionView): Promise<KnowledgeCollectionRow> => {
const { documents } = await caller.call(documentList, { ...slugs, collectionName: collection.name })
return { name: collection.name, documents }
}),
)
return {
title: t("title"),
intro: t("intro"),
status: {
enabled: status.enabled,
labels: statusLabels(t, status),
state: serviceStateLabel(t, status.service.state),
reasons: status.service.reasons,
...(status.service.refusal === undefined
? {}
: {
refusal: {
code: status.service.refusal,
text: refusalLabels(t)[status.service.refusal],
},
}),
},
collections: rows,
documents: {
caption: t("documents.heading"),
file: t("documents.file"),
status: t("documents.status"),
error: t("documents.error"),
statuses: Object.fromEntries(
Object.entries(documentStatusKey).map(([status, key]) => [status, t(key)]),
) as DocumentListLabels["statuses"],
failures: failureLabels(t),
},
index: indexFormLabels(t),
search: searchFormLabels(t),
autoIndex: {
current: auto.collectionName,
labels: {
heading: t("autoIndex.heading"),
hint: t("autoIndex.hint"),
label: t("autoIndex.label"),
off: t("autoIndex.off"),
submit: t("autoIndex.submit"),
busy: t("autoIndex.busy"),
done: t("autoIndex.done"),
},
},
}
},
View({ slugs, locale, data }: ProjectPageViewProps<KnowledgePageData>): ReactElement {
const { labels, enabled, state, reasons, refusal } = data.status
return (
<Page>
<PageHeader title={data.title} description={data.intro} />
<PageSection title={labels.heading} variant="card">
{refusal ? (
<Alert variant="warning" role="status" data-state={refusal.code}>
<AlertDescription>{refusal.text}</AlertDescription>
</Alert>
) : null}
{enabled ? (
<FactList
facts={[
{ label: labels.state, value: state },
...(reasons.length > 0
? [
{
label: labels.reasons,
value: (
<ul>
{reasons.map((reason) => (
<li key={reason}>{reason}</li>
))}
</ul>
),
},
]
: []),
{ label: labels.indexing, value: labels.documents },
]}
/>
) : (
<p role="status">{labels.flagOff}</p>
)}
</PageSection>
{enabled && data.collections.length > 0 ? (
<AutoIndexForm
slugs={slugs}
collections={data.collections.map((collection) => collection.name)}
current={data.autoIndex.current}
labels={data.autoIndex.labels}
/>
) : null}
{enabled
? data.collections.map((collection) => (
<PageSection key={collection.name} title={collection.name}>
<IndexForm slugs={slugs} collectionName={collection.name} labels={data.index} />
<SearchForm slugs={slugs} collectionName={collection.name} labels={data.search} locale={locale} />
<CollectionDocuments
slugs={slugs}
collectionName={collection.name}
seed={collection.documents}
labels={data.documents}
/>
</PageSection>
))
: null}
</Page>
)
},
})
+91
View File
@@ -0,0 +1,91 @@
"use client"
// The search form: one question, cited answers. The disclosure the EU AI Act Article 50 asks for
// rides with the results, and every answer names its source and chunk — a person checks a passage
// against the file, never a model's word.
import { formatMessage } from "@conusai-kit/i18n/format"
import { AiDisclosureBadge, type ProjectSlugs } from "@conusai-kit/ui"
import { useDataClient } from "@conusai-kit/ui/data"
import { FormLayout, type FormValues, PageSection, TextField } from "@conusai-kit/ui/patterns"
import { Alert, AlertDescription } from "@conusai-kit/ui/primitives"
import { type ReactElement, useState } from "react"
import type { SearchResult } from "../contract/index.js"
import { type RefusalNotice, refusalOf, type SearchFormLabels } from "./labels.js"
export function SearchForm({
slugs,
collectionName,
labels,
locale,
}: {
slugs: ProjectSlugs
collectionName: string
labels: SearchFormLabels
locale: string
}): ReactElement {
const client = useDataClient()
const [results, setResults] = useState<SearchResult[] | null>(null)
const [refused, setRefused] = useState<RefusalNotice | undefined>(undefined)
const search = async ({ query = "" }: FormValues): Promise<void> => {
setResults(null)
setRefused(undefined)
try {
const { value: answer } = await client.call<{ results: SearchResult[] }>("knowledge.search", {
...slugs,
collectionName,
query,
})
setResults(answer.results)
} catch (error) {
// A refusal of the service has its own words. Anything else is FormLayout's to show.
const notice = refusalOf(labels.refusals, error)
if (notice === undefined) throw error
setRefused(notice)
}
}
return (
<PageSection title={labels.heading} level={3} variant="card">
<AiDisclosureBadge label={labels.disclosure} />
<FormLayout submitLabel={labels.submit} busyLabel={labels.busy} onSubmit={search}>
<TextField
name="query"
type="search"
label={labels.label}
placeholder={labels.placeholder}
required
maxLength={500}
/>
</FormLayout>
{refused ? (
<Alert variant="warning" data-state={refused.code}>
<AlertDescription>{refused.text}</AlertDescription>
</Alert>
) : null}
{results === null ? null : results.length === 0 ? (
<p>{labels.empty}</p>
) : (
<ol aria-label={formatMessage(labels.results, { count: results.length }, locale)}>
{results.map((result, at) => (
<li key={`${result.fileId}-${result.chunk}-${at}`}>
<blockquote
data-knowledge-chunk={result.chunkId}
data-source-page={result.page}
data-source-block={result.blockId}
>
{result.snippet}
</blockquote>
<p>
{labels.source}: {result.filename} · {labels.chunk.replace("{chunk}", String(result.chunk))}
{result.page === undefined ? null : ` · ${labels.page.replace("{page}", String(result.page))}`}
{result.blockId ? <code> {result.blockId}</code> : null}
</p>
</li>
))}
</ol>
)}
</PageSection>
)
}
+106
View File
@@ -0,0 +1,106 @@
"use client"
import { Button } from "@conusai-kit/ui/primitives"
// The enablement controls of the organization's settings page: the current state, the enable
// button with its typed refusal reasons (plan, policy, quota), and disable — which preserves
// every collection, document and the service's index.
import { describeError, useWrite } from "@conusai-kit/ui/data"
import { PageSection } from "@conusai-kit/ui/patterns"
import { type ReactElement, useState } from "react"
export interface SettingsSectionLabels {
readonly heading: string
readonly state: string
readonly stateOn: string
readonly stateOff: string
readonly backend: string
readonly backendPlatform: string
readonly enable: string
readonly enabling: string
readonly disable: string
readonly disabling: string
readonly enabledDone: string
readonly disabledDone: string
readonly planRequired: string
readonly featureDisabled: string
readonly quotaExhausted: string
readonly upgrade: string
readonly hint: string
readonly failed: string
}
type View = { kind: "idle" } | { kind: "busy" } | { kind: "message"; text: string; alert: boolean }
export function SettingsSection({
org,
initial,
labels,
}: {
org: string
initial: { enabled: boolean; backend: string | null }
labels: SettingsSectionLabels
}): ReactElement {
const enableAi = useWrite<{ org: string }, unknown>("knowledge.settings.enableDocumentAi")
const disableAi = useWrite<{ org: string }, unknown>("knowledge.settings.disableDocumentAi")
const [enabled, setEnabled] = useState(initial.enabled)
const [view, setView] = useState<View>({ kind: "idle" })
const enable = async (): Promise<void> => {
setView({ kind: "busy" })
try {
await enableAi.mutate({ org })
setEnabled(true)
setView({ kind: "message", text: labels.enabledDone, alert: false })
} catch (error) {
const code = (error as { problem?: { code?: string } } | null)?.problem?.code ?? "unknown"
if (code === "plan_required") {
setView({ kind: "message", text: `${labels.planRequired} ${labels.upgrade}`, alert: true })
} else if (code === "quota_exhausted") {
setView({ kind: "message", text: labels.quotaExhausted, alert: true })
} else if (code === "feature_disabled") {
setView({ kind: "message", text: labels.featureDisabled, alert: true })
} else {
setView({ kind: "message", text: describeError(error), alert: true })
}
}
}
const disable = async (): Promise<void> => {
setView({ kind: "busy" })
try {
await disableAi.mutate({ org })
setEnabled(false)
setView({ kind: "message", text: labels.disabledDone, alert: false })
} catch (error) {
setView({ kind: "message", text: describeError(error), alert: true })
}
}
return (
<PageSection title={labels.heading}>
<p>{labels.hint}</p>
<dl>
<dt>{labels.state}</dt>
<dd>{enabled ? labels.stateOn : labels.stateOff}</dd>
{initial.backend !== null ? (
<>
<dt>{labels.backend}</dt>
<dd>{labels.backendPlatform}</dd>
</>
) : null}
</dl>
{view.kind === "message" ? <p role={view.alert ? "alert" : "status"}>{view.text}</p> : null}
{enabled ? (
<Button type="button" disabled={view.kind === "busy"} onClick={() => void disable()}>
{view.kind === "busy" ? labels.disabling : labels.disable}
</Button>
) : (
<Button type="button" disabled={view.kind === "busy"} onClick={() => void enable()}>
{view.kind === "busy" ? labels.enabling : labels.enable}
</Button>
)}
</PageSection>
)
}