@@ -0,0 +1,124 @@
|
||||
# knowledge
|
||||
|
||||
The retrieval module: named collections of the app's documents, an idempotent document-AI
|
||||
enablement flow (`knowledge.settings.enableDocumentAi`), durable index/remove jobs, and cited
|
||||
search over the platform's knowledge service.
|
||||
|
||||
## Events
|
||||
|
||||
A web request writes the document row and publishes `knowledge.documentIndexRequested` or
|
||||
`knowledge.documentRemoveRequested` in its transaction; the module's subscribers, run by the worker's
|
||||
outbox relay, turn each into the `knowledge.index` or `knowledge.remove` job. A deleted file
|
||||
(`files.FileDeleted`) cancels every document that indexed it and asks for its removal, carrying the storage key the
|
||||
file had: its `files` row is gone by then, so the `knowledge.remove` job removes the platform document by that key's
|
||||
Blob pathname rather than looking the file up (F407). A removal a person asks for reads the stored file's key when
|
||||
it is asked for, so a file deleted before the job runs is still removed by that key; a person's removal of a file
|
||||
already being deleted is refused (409), and the deletion's own removal does it (F407, review R1). The relay delivers
|
||||
in the organization's scope with a system actor, so a subscriber takes the organization from the delivery
|
||||
and only the workspace and project from the record, and refuses a record of another organization
|
||||
(`adapters/jobs.ts`). `test/relay-scope.test.ts` runs these deliveries through the worker's own relay on
|
||||
PostgreSQL.
|
||||
|
||||
## The search backend
|
||||
|
||||
Retrieval is one platform service (ADR-061): an app keeps no vector store and no embedding pipeline
|
||||
of its own. The web shell and the worker bind one `KnowledgeSearchService` adapter, `platform`
|
||||
(`KNOWLEDGE_BACKEND`, the default and the only value; anything else stops the app at start, by the
|
||||
variable's name):
|
||||
|
||||
- The adapter calls the platform's managed documents-index service through the kit-platform
|
||||
`Knowledge` client, with a deployment token that holds `knowledge.collections:use` (`conusai.json` asks for it,
|
||||
in every profile's copy that has a token block; `*` never carries it). The service's
|
||||
`document_ref` is the object's pathname inside the project's Blob prefix and
|
||||
`options.content_sha256` is required. The adapter hashes the job's bytes, resolves that pathname
|
||||
through the files module's scoped storage reference, and maps returned citations back through the
|
||||
same project scope. Foreign or stale references are omitted; public file views still do not expose
|
||||
storage keys. The service keeps a collection per platform project and environment and answers 404 for
|
||||
one it does not hold, so the first index creates it in `CONUSAI_ENVIRONMENT`, and a search of a
|
||||
collection nothing was indexed into answers no results.
|
||||
- A refusal of the service is one closed set of codes, `KNOWLEDGE_REFUSALS`
|
||||
(`adapters/refusals.ts`): `plan_required`, `feature_disabled`, `quota_exhausted`,
|
||||
`budget_exhausted`, `scope_missing` (the token lacks `knowledge.collections:use`), `indexing`,
|
||||
`approval_required`, `runtime_unavailable`. `knowledge.status` carries it as `service.refusal`,
|
||||
`knowledge.search` answers it as the problem's `code`, and the pages show `knowledge.refusals.<code>`
|
||||
in English or Lithuanian. An index job records one that waiting cannot fix on the document's row
|
||||
instead of retrying it.
|
||||
|
||||
The module is off for an organization until an administrator enables document AI: with nothing bound
|
||||
to `KNOWLEDGE_FLAGS`, the `knowledge.enabled` flag answers the organization's `knowledge_config` row.
|
||||
|
||||
The local `vectors` backend (a linked Qdrant or pgvector, embeddings through the AI gateway) was
|
||||
removed: no profile needs it, since this module runs on `postgres` alone and ADR-061 gives retrieval to
|
||||
the platform. `knowledge_config` keeps the columns that backend used (`source`, `service_id`, `url`):
|
||||
the migration has run, and nothing reads them.
|
||||
|
||||
## The live document list
|
||||
|
||||
The module declares the live collection `knowledge.document` (`contract/collections.ts`: table
|
||||
`knowledge_document`, key `documentId`, batch-get `knowledge.document.getMany` under
|
||||
`knowledge.knowledge.search`, at most 100 ids). Each collection's list on the page keeps the loader's
|
||||
rows as the seed of a `liveCollection` (`ui/live-documents.tsx`), filtered by the collection's name since
|
||||
a live topic is the whole project. A document another member asks to index appears, and each moves from
|
||||
queued to indexed or failed as the worker and the platform answer. Indexing is a `useWrite` that
|
||||
resolves once the live stream has seen the row; nothing re-runs the page's loader after a write
|
||||
(`test/live-writes.test.ts`).
|
||||
|
||||
## Automatic PDF indexing (core-11)
|
||||
|
||||
An administrator can opt a project in on its knowledge page (`knowledge.autoindex.set`, permission
|
||||
`knowledge.knowledge.manage`), naming an existing collection. The opt-in is one row of
|
||||
`knowledge_auto_index` (migration `0002`) that records who turned it on; no row is off.
|
||||
|
||||
The module contributes a `files.processors` processor for `application/pdf`, so the files module
|
||||
enqueues `knowledge.autoindex` once per stored PDF, in the upload's project (`adapters/jobs.ts`). The
|
||||
job reads nothing for a project that is not opted in, has knowledge off, or whose file is gone or no
|
||||
longer has the upload's digest. Otherwise it makes a `knowledge_document` row requested by the recorded
|
||||
administrator and runs the index lifecycle with that row (`core/index-job.ts`, `indexPassages`):
|
||||
|
||||
- The job runs for nobody, so the worker's native admission checks the administrator's
|
||||
`knowledge.knowledge.manage` and the platform's OCR and knowledge services again before each step
|
||||
that leaves the process (`apps/worker/src/native-admission.ts`).
|
||||
- Before the read is paid for, the organization's eligibility and indexing quota are checked again
|
||||
through `KNOWLEDGE_ENABLEMENT`, as the enablement flow checks them. A refusal is recorded as
|
||||
`plan_required`, `feature_disabled` or `quota_exhausted`. There is no platform pre-check API for the
|
||||
quota yet, so today the knowledge service's own refusal at call time is the hard limit, as it is for a
|
||||
person's manual index.
|
||||
- The platform's document reader reads the PDF once per file and digest, and `@conusai-sdk/ocr/passages`
|
||||
cuts it into passages of at most the service's 64 KiB per item (`adapters/reader.ts`). A failed read,
|
||||
or a block id no metadata value can cite, is recorded as `not_indexable`.
|
||||
- Each passage is one inline item of the platform's knowledge service, under the file's Blob reference
|
||||
plus `#p{page}.{index}`, in the project's tenant partition, with metadata `page`, `block_ids` and
|
||||
`file_id` (`adapters/platform-knowledge.ts`, `platformPassageIndex`). A search hit on a passage cites
|
||||
its file, its page and its first block.
|
||||
- Before anything is sent, the file's reference and the count of passages per page are recorded in
|
||||
`knowledge_passage_set` (migration `0003`; references and counts only, no text). Removing the
|
||||
document, as a deleted file does, tombstones every passage under that reference, even after the
|
||||
file itself is gone or the service refused the run partway.
|
||||
|
||||
The worker binds `KNOWLEDGE_READER` and `KNOWLEDGE_PASSAGES` lazily; the web shell binds neither, as a
|
||||
request never reads a PDF.
|
||||
|
||||
## Why the module is Postgres-only (no Convex profile yet)
|
||||
|
||||
`module.convex.ts` ships a fail-closed Convex binding for a future wiring, but the module is
|
||||
**not** registered in the Convex profile (`convex/lib/runtime.ts`), and the manifest stays
|
||||
`profiles: ["postgres"]`. The integration review (2026-10-23) kept it Postgres-only because the
|
||||
module genuinely cannot run on the Convex runtime today:
|
||||
|
||||
1. `knowledge.search` is an **effect** procedure — every backend leaves the process (the managed
|
||||
service over HTTP, or embeddings through the AI gateway plus a vector store). Convex
|
||||
queries/mutations cannot perform network I/O; effects need an explicit action composition like
|
||||
the files module's action→mutation workflows, and none exists for search.
|
||||
2. The index/remove **jobs** must run where the platform SDKs can leave the process. The Convex
|
||||
binding fails closed (`… must run through the Convex Node action boundary`); unlike documents'
|
||||
`documentsActions:process`, no composed Node action exists for these jobs yet, so every
|
||||
indexing request on the Convex profile would dispatch straight into the fence.
|
||||
3. `knowledge.settings.enableDocumentAi` is an `effect`: its use case performs network I/O
|
||||
(eligibility, quota), impossible inside a Convex mutation, so it needs
|
||||
its own action composition.
|
||||
4. `knowledge.status` takes its readiness snapshot from `KNOWLEDGE_READINESS`, which the Convex
|
||||
container does not bind, so it would answer `unavailable`.
|
||||
|
||||
Wiring the profile means composing those Node actions first (mirror `convex/documentsActions.ts`);
|
||||
registering the module before then would ship a broken search, a broken enablement flow and
|
||||
always-failing jobs, not a fail-closed gate.
|
||||
@@ -0,0 +1,62 @@
|
||||
import { Container, type TenantScope } from "@conusai-kit/core"
|
||||
import { createMemoryDataSource, DATA_SOURCE, SYSTEM_DATA } from "@conusai-kit/data"
|
||||
import { accountData, projects } from "@conusai-kit/tenancy"
|
||||
import { expect, it } from "vitest"
|
||||
import { custodyModule } from "../../custody.js"
|
||||
import { knowledgeAccountData } from "../adapters/account.js"
|
||||
import { autoIndex, KNOWLEDGE_ENTITIES } from "../contract/index.js"
|
||||
import { knowledgeModule } from "../module.js"
|
||||
|
||||
const subject = "01900000-0000-7000-0000-000000000009"
|
||||
async function fixture() {
|
||||
const data = createMemoryDataSource({ entities: [projects, ...KNOWLEDGE_ENTITIES] })
|
||||
const container = new Container()
|
||||
container.value(DATA_SOURCE, data)
|
||||
container.value(SYSTEM_DATA, data)
|
||||
const scopes: TenantScope[] = []
|
||||
for (const n of [1, 2, 3]) {
|
||||
const orgId = `01900000-0000-7000-0000-00000000000${n}`
|
||||
const workspaceId = `01900000-0000-7000-0000-00000000001${n}`
|
||||
const actor = { userId: subject }
|
||||
const project = await data
|
||||
.forScope({ level: "workspace", orgId, workspaceId, actor })
|
||||
.of(projects)
|
||||
.insert({ slug: "site", name: "Site", moduleKey: "notes", archivedAt: null })
|
||||
const scope: TenantScope = { level: "project", orgId, workspaceId, projectId: project.id, actor }
|
||||
scopes.push(scope)
|
||||
await data
|
||||
.forScope(scope)
|
||||
.of(autoIndex)
|
||||
.insert({
|
||||
collectionName: "contracts",
|
||||
enabledBy: n === 3 ? "another-person" : subject,
|
||||
enabledAt: "2026-11-11T00:00:00.000Z",
|
||||
})
|
||||
}
|
||||
return { data, container, scopes }
|
||||
}
|
||||
|
||||
it("exports and erases only that administrator’s opt-ins across projects, idempotently", async () => {
|
||||
const { data, container, scopes } = await fixture()
|
||||
const exported = await knowledgeAccountData.export?.({ subject, container })
|
||||
expect(exported?.automaticIndexing).toHaveLength(2)
|
||||
await knowledgeAccountData.erase?.({ subject, container })
|
||||
expect(await knowledgeAccountData.export?.({ subject, container })).toEqual({ automaticIndexing: [] })
|
||||
await knowledgeAccountData.erase?.({ subject, container })
|
||||
const otherScope = scopes[2]
|
||||
if (!otherScope) throw new Error("Missing unrelated project fixture")
|
||||
expect((await data.forScope(otherScope).of(autoIndex).list()).items).toHaveLength(1)
|
||||
})
|
||||
|
||||
it("retains the account contribution when processing is deselected", () => {
|
||||
const retained = custodyModule(knowledgeModule)
|
||||
expect(retained.extends?.find((entry) => entry.point.id === accountData.id)?.values).toEqual([knowledgeAccountData])
|
||||
})
|
||||
|
||||
it("an aborted account operation changes no opt-in", async () => {
|
||||
const { data, container, scopes } = await fixture()
|
||||
const controller = new AbortController()
|
||||
controller.abort()
|
||||
await expect(knowledgeAccountData.erase?.({ subject, container, signal: controller.signal })).rejects.toThrow()
|
||||
for (const scope of scopes) expect((await data.forScope(scope).of(autoIndex).list()).items).toHaveLength(1)
|
||||
})
|
||||
@@ -0,0 +1,309 @@
|
||||
// The automatic PDF intake (template lesson core-11): an uploaded PDF of an opted-in project is read by
|
||||
// the platform's document reader and indexed passage by passage into the opted-in collection, for the
|
||||
// administrator who turned it on, whose permission is checked again before each step that leaves the
|
||||
// process. A project that is not opted in, or has knowledge off, reads nothing; removing the document
|
||||
// tombstones every passage it indexed, even once the file itself is gone.
|
||||
import type { TenantScope } from "@conusai-kit/core"
|
||||
import { createMemoryDataSource } from "@conusai-kit/data"
|
||||
import { DEFAULT_FILES_POLICY, files, filesProcessors, sha256Hex } from "@conusai-kit/files"
|
||||
import { fakeObjectStore } from "@conusai-kit/testing"
|
||||
import { describe, expect, it } from "vitest"
|
||||
import {
|
||||
AUTO_INDEX_JOB,
|
||||
AUTO_INDEX_PROCESSOR_KEY,
|
||||
autoIndex,
|
||||
collections,
|
||||
documents,
|
||||
KNOWLEDGE_ENTITIES,
|
||||
passageSets,
|
||||
} from "../contract/index.js"
|
||||
import { autoIndexUpload } from "../core/auto-index-job.js"
|
||||
import { removeDocument } from "../core/index-job.js"
|
||||
import type {
|
||||
DocumentAiPlatform,
|
||||
KnowledgeDeps,
|
||||
KnowledgePassage,
|
||||
KnowledgePassageIndex,
|
||||
KnowledgeReader,
|
||||
} from "../core/ports.js"
|
||||
import { knowledgeModule } from "../module.js"
|
||||
|
||||
const ADMIN = "01900000-0000-7000-0000-000000000009"
|
||||
const UPLOADER = "01900000-0000-7000-0000-000000000010"
|
||||
const scope: TenantScope = {
|
||||
level: "project",
|
||||
actor: { userId: "system:knowledge.autoindex", system: true },
|
||||
orgId: "01900000-0000-7000-0000-000000000001",
|
||||
workspaceId: "01900000-0000-7000-0000-000000000002",
|
||||
projectId: "01900000-0000-7000-0000-000000000003",
|
||||
}
|
||||
const pdf = new TextEncoder().encode("%PDF-1.7 synthetic")
|
||||
const read: readonly KnowledgePassage[] = [
|
||||
{ page: 1, index: 0, blockIds: ["b1", "b2"], text: "First page, first passage" },
|
||||
{ page: 1, index: 1, blockIds: ["b3"], text: "First page, second passage" },
|
||||
{ page: 3, index: 0, blockIds: [], text: "Third page" },
|
||||
]
|
||||
|
||||
function makeDeps(
|
||||
options: { enabled?: boolean; reader?: KnowledgeReader; indexError?: Error; enablement?: DocumentAiPlatform } = {},
|
||||
) {
|
||||
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES, files] })
|
||||
const store = fakeObjectStore()
|
||||
const calls = {
|
||||
reads: [] as Array<{ filename: string; idempotencyKey: string }>,
|
||||
indexed: [] as Array<{ collectionName: string; fileId: string; passages: readonly KnowledgePassage[] }>,
|
||||
removed: [] as Array<{ documentRef: string; passages: ReadonlyArray<{ page: number; index: number }> }>,
|
||||
admitted: [] as string[],
|
||||
}
|
||||
const reader: KnowledgeReader = options.reader ?? {
|
||||
async read(bytes, input) {
|
||||
expect(bytes).toEqual(pdf)
|
||||
calls.reads.push({ filename: input.filename, idempotencyKey: input.idempotencyKey })
|
||||
return read
|
||||
},
|
||||
}
|
||||
const passages: KnowledgePassageIndex = {
|
||||
async reference(_scope, fileId) {
|
||||
return `org/${scope.orgId}/${fileRow.storageKey}?${fileId}`
|
||||
},
|
||||
async index(_scope, input) {
|
||||
expect(input.documentRef).toBe(`org/${scope.orgId}/${fileRow.storageKey}?${input.fileId}`)
|
||||
calls.indexed.push({ collectionName: input.collectionName, fileId: input.fileId, passages: input.passages })
|
||||
if (options.indexError) throw options.indexError
|
||||
},
|
||||
async remove(_scope, input) {
|
||||
calls.removed.push({ documentRef: input.documentRef, passages: input.passages })
|
||||
},
|
||||
}
|
||||
const deps: KnowledgeDeps = {
|
||||
data,
|
||||
files: { data, store, policy: DEFAULT_FILES_POLICY },
|
||||
flags: { enabled: async () => options.enabled ?? true },
|
||||
reader,
|
||||
passages,
|
||||
enablement: options.enablement ?? {
|
||||
checkEligibility: async () => ({ ok: true }),
|
||||
indexingQuota: async () => 1000,
|
||||
},
|
||||
beforeIndex: async (subject) => {
|
||||
calls.admitted.push(subject)
|
||||
},
|
||||
}
|
||||
return { data, store, deps, calls }
|
||||
}
|
||||
|
||||
const fileRow = {
|
||||
status: "stored" as const,
|
||||
storageKey: `files/${scope.workspaceId}/${scope.projectId}/01900000-0000-7000-0000-000000000100`,
|
||||
filename: "contract.pdf",
|
||||
contentType: "application/pdf",
|
||||
size: pdf.length,
|
||||
sha256: "",
|
||||
uploadedBy: UPLOADER,
|
||||
storedAt: "2026-11-09T19:00:00.000Z",
|
||||
uploadId: null,
|
||||
declaredSize: null,
|
||||
partSize: null,
|
||||
deletingAt: null,
|
||||
}
|
||||
|
||||
async function seed(fixture: ReturnType<typeof makeDeps>, options: { optIn?: boolean } = {}) {
|
||||
const sha256 = await sha256Hex(pdf)
|
||||
await fixture.data.forScope(scope).of(collections).insert({ name: "contracts", createdBy: ADMIN })
|
||||
if (options.optIn ?? true)
|
||||
await fixture.data
|
||||
.forScope(scope)
|
||||
.of(autoIndex)
|
||||
.insert({ collectionName: "contracts", enabledBy: ADMIN, enabledAt: "2026-11-09T18:00:00.000Z" })
|
||||
const file = await fixture.data
|
||||
.forScope(scope)
|
||||
.of(files)
|
||||
.insert({ ...fileRow, sha256 })
|
||||
await fixture.store.put(scope, { key: fileRow.storageKey, body: pdf, contentType: "application/pdf" })
|
||||
return { fileId: file.id, storageKey: fileRow.storageKey, contentType: "application/pdf", sha256 }
|
||||
}
|
||||
|
||||
/** The run's document, which these cases expect there to be. */
|
||||
async function indexedView<T>(view: T | null): Promise<T> {
|
||||
if (view === null) throw new Error("the upload made no document")
|
||||
return view
|
||||
}
|
||||
|
||||
describe("the automatic PDF intake", () => {
|
||||
it("reads an opted-in project's PDF and indexes its passages into the collection, for the administrator who opted in", async () => {
|
||||
const fixture = makeDeps()
|
||||
const upload = await seed(fixture)
|
||||
|
||||
const done = await autoIndexUpload(fixture.deps, scope, upload)
|
||||
|
||||
expect(done).toMatchObject({ collectionName: "contracts", fileId: upload.fileId, status: "indexed" })
|
||||
expect(fixture.calls.reads).toEqual([
|
||||
{ filename: "contract.pdf", idempotencyKey: `knowledge.autoindex.${upload.fileId}.${upload.sha256}` },
|
||||
])
|
||||
expect(fixture.calls.indexed).toEqual([{ collectionName: "contracts", fileId: upload.fileId, passages: read }])
|
||||
// The job runs for nobody else: the administrator's permission is what each step leaving the process checks.
|
||||
expect(fixture.calls.admitted.length).toBeGreaterThan(0)
|
||||
expect(new Set(fixture.calls.admitted)).toEqual(new Set([ADMIN]))
|
||||
const [row] = (await fixture.data.forScope(scope).of(documents).list({ limit: 10 })).items
|
||||
expect(row).toMatchObject({ filename: "contract.pdf", requestedBy: ADMIN, status: "indexed", error: null })
|
||||
// What removal needs, kept beside the document: the file's reference and the passages per page.
|
||||
const [set] = (await fixture.data.forScope(scope).of(passageSets).list({ limit: 10 })).items
|
||||
expect(set).toMatchObject({
|
||||
collectionName: "contracts",
|
||||
fileId: upload.fileId,
|
||||
documentRef: `org/${scope.orgId}/${fileRow.storageKey}?${upload.fileId}`,
|
||||
pages: "2,0,1",
|
||||
})
|
||||
})
|
||||
|
||||
it("is the same work once: a redelivered job reads and indexes nothing again", async () => {
|
||||
const fixture = makeDeps()
|
||||
const upload = await seed(fixture)
|
||||
await autoIndexUpload(fixture.deps, scope, upload)
|
||||
expect(await autoIndexUpload(fixture.deps, scope, upload)).toMatchObject({ status: "indexed" })
|
||||
expect(fixture.calls.reads).toHaveLength(1)
|
||||
expect(fixture.calls.indexed).toHaveLength(1)
|
||||
})
|
||||
|
||||
it("reads nothing for a project that is not opted in, has knowledge off, or a file that is not a PDF", async () => {
|
||||
const notOptedIn = makeDeps()
|
||||
expect(await autoIndexUpload(notOptedIn.deps, scope, await seed(notOptedIn, { optIn: false }))).toBeNull()
|
||||
|
||||
const off = makeDeps({ enabled: false })
|
||||
expect(await autoIndexUpload(off.deps, scope, await seed(off))).toBeNull()
|
||||
|
||||
const text = makeDeps()
|
||||
const upload = await seed(text)
|
||||
expect(await autoIndexUpload(text.deps, scope, { ...upload, contentType: "text/plain" })).toBeNull()
|
||||
|
||||
for (const fixture of [notOptedIn, off, text]) {
|
||||
expect(fixture.calls.reads).toEqual([])
|
||||
expect((await fixture.data.forScope(scope).of(documents).list({ limit: 10 })).items).toEqual([])
|
||||
}
|
||||
})
|
||||
|
||||
it("reads nothing once the administrator's permission is gone: the refusal is the job's", async () => {
|
||||
const fixture = makeDeps()
|
||||
const upload = await seed(fixture)
|
||||
const refused = Object.assign(new Error("permission_missing"), { status: 403 })
|
||||
const deps: KnowledgeDeps = {
|
||||
...fixture.deps,
|
||||
beforeIndex: async () => {
|
||||
throw refused
|
||||
},
|
||||
}
|
||||
await expect(autoIndexUpload(deps, scope, upload)).rejects.toBe(refused)
|
||||
expect(fixture.calls.reads).toEqual([])
|
||||
expect(fixture.calls.indexed).toEqual([])
|
||||
})
|
||||
|
||||
it("records a PDF the reader cannot make passages of as failed, with a stable code", async () => {
|
||||
const fixture = makeDeps({
|
||||
reader: {
|
||||
read: async () => {
|
||||
throw Object.assign(new Error("block id over 256 characters"), { permanent: "not_indexable" })
|
||||
},
|
||||
},
|
||||
})
|
||||
const upload = await seed(fixture)
|
||||
expect(await autoIndexUpload(fixture.deps, scope, upload)).toMatchObject({
|
||||
status: "failed",
|
||||
error: "not_indexable",
|
||||
})
|
||||
expect(fixture.calls.indexed).toEqual([])
|
||||
})
|
||||
|
||||
it("keeps the passage layout of an index the service refused partway, so removal still tombstones it", async () => {
|
||||
const fixture = makeDeps({ indexError: Object.assign(new Error("quota"), { permanent: "quota_exhausted" }) })
|
||||
const upload = await seed(fixture)
|
||||
const failed = await autoIndexUpload(fixture.deps, scope, upload)
|
||||
expect(failed).toMatchObject({ status: "failed", error: "quota_exhausted" })
|
||||
await removeDocument(fixture.deps, scope, (await indexedView(failed)).documentId)
|
||||
expect(fixture.calls.removed.map((call) => call.passages.length)).toEqual([3])
|
||||
expect((await fixture.data.forScope(scope).of(passageSets).list({ limit: 10 })).items).toEqual([])
|
||||
})
|
||||
|
||||
it("reads nothing once the organization is no longer eligible or its indexing quota is spent", async () => {
|
||||
const ineligible = makeDeps({
|
||||
enablement: {
|
||||
checkEligibility: async () => ({
|
||||
ok: false,
|
||||
reason: "feature_disabled",
|
||||
detail: "the agent_document_tools flag is off",
|
||||
}),
|
||||
indexingQuota: async () => 1000,
|
||||
},
|
||||
})
|
||||
expect(await autoIndexUpload(ineligible.deps, scope, await seed(ineligible))).toMatchObject({
|
||||
status: "failed",
|
||||
error: "feature_disabled",
|
||||
})
|
||||
|
||||
const spent = makeDeps({
|
||||
enablement: { checkEligibility: async () => ({ ok: true }), indexingQuota: async () => 0 },
|
||||
})
|
||||
expect(await autoIndexUpload(spent.deps, scope, await seed(spent))).toMatchObject({
|
||||
status: "failed",
|
||||
error: "quota_exhausted",
|
||||
})
|
||||
|
||||
for (const fixture of [ineligible, spent]) {
|
||||
expect(fixture.calls.reads).toEqual([])
|
||||
expect(fixture.calls.indexed).toEqual([])
|
||||
}
|
||||
})
|
||||
|
||||
it("refuses to run where the enablement platform is not bound, rather than skip its checks", async () => {
|
||||
const fixture = makeDeps()
|
||||
const upload = await seed(fixture)
|
||||
const { enablement: _unbound, ...deps } = fixture.deps
|
||||
await expect(autoIndexUpload(deps, scope, upload)).rejects.toThrow("enablement")
|
||||
expect(fixture.calls.reads).toEqual([])
|
||||
})
|
||||
|
||||
it("does not read a file whose bytes changed since the upload", async () => {
|
||||
const fixture = makeDeps()
|
||||
const upload = await seed(fixture)
|
||||
expect(await autoIndexUpload(fixture.deps, scope, { ...upload, sha256: "0".repeat(64) })).toBeNull()
|
||||
expect(fixture.calls.reads).toEqual([])
|
||||
})
|
||||
|
||||
it("tombstones every passage it indexed when the document is removed, even after the file is gone", async () => {
|
||||
const fixture = makeDeps()
|
||||
const upload = await seed(fixture)
|
||||
const done = await autoIndexUpload(fixture.deps, scope, upload)
|
||||
await fixture.data.forScope(scope).of(files).delete(upload.fileId)
|
||||
|
||||
await removeDocument(fixture.deps, scope, (await indexedView(done)).documentId)
|
||||
|
||||
expect(fixture.calls.removed).toEqual([
|
||||
{
|
||||
documentRef: `org/${scope.orgId}/${fileRow.storageKey}?${upload.fileId}`,
|
||||
passages: [
|
||||
{ page: 1, index: 0 },
|
||||
{ page: 1, index: 1 },
|
||||
{ page: 3, index: 0 },
|
||||
],
|
||||
},
|
||||
])
|
||||
expect((await fixture.data.forScope(scope).of(documents).list({ limit: 10 })).items).toEqual([])
|
||||
expect((await fixture.data.forScope(scope).of(passageSets).list({ limit: 10 })).items).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe("the intake's place in the manifest", () => {
|
||||
it("contributes a files processor for PDFs whose job the module runs", () => {
|
||||
const contributions = (knowledgeModule.extends ?? []).filter((entry) => entry.point.id === filesProcessors.id)
|
||||
expect(contributions.flatMap((entry) => entry.values)).toEqual([
|
||||
{ key: AUTO_INDEX_PROCESSOR_KEY, contentTypes: ["application/pdf"], job: AUTO_INDEX_JOB },
|
||||
])
|
||||
expect(Object.keys(knowledgeModule.jobs ?? {})).toContain(AUTO_INDEX_JOB)
|
||||
})
|
||||
|
||||
it("declares the platform operations the job uses: the knowledge service's and the document reader's", () => {
|
||||
const operations = (service: string) =>
|
||||
knowledgeModule.serviceRequirements?.find((requirement) => requirement.service === service)?.operations
|
||||
expect(operations("knowledge")?.[AUTO_INDEX_JOB]).toEqual(["knowledge.collections:use"])
|
||||
expect(operations("ocr")?.[AUTO_INDEX_JOB]).toEqual(["ocr:use"])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,73 @@
|
||||
// A project's automatic PDF indexing (template lesson core-11): an administrator turns it on into an
|
||||
// existing collection, and is recorded as the one whose permission each indexing job re-checks; off
|
||||
// is no row. Another project's collection is not found, and a project with knowledge off refuses.
|
||||
import type { TenantScope } from "@conusai-kit/core"
|
||||
import { createMemoryDataSource } from "@conusai-kit/data"
|
||||
import { DEFAULT_FILES_POLICY, files } from "@conusai-kit/files"
|
||||
import { fakeObjectStore } from "@conusai-kit/testing"
|
||||
import { describe, expect, it } from "vitest"
|
||||
import { autoIndex, collections, KNOWLEDGE_ENTITIES } from "../contract/index.js"
|
||||
import { getAutoIndex, setAutoIndex } from "../core/auto-index.js"
|
||||
import type { KnowledgeDeps } from "../core/ports.js"
|
||||
|
||||
const ADMIN = "01900000-0000-7000-0000-000000000009"
|
||||
const scope: TenantScope = {
|
||||
level: "project",
|
||||
actor: { userId: ADMIN },
|
||||
orgId: "01900000-0000-7000-0000-000000000001",
|
||||
workspaceId: "01900000-0000-7000-0000-000000000002",
|
||||
projectId: "01900000-0000-7000-0000-000000000003",
|
||||
}
|
||||
const otherProject: TenantScope = { ...scope, projectId: "01900000-0000-7000-0000-000000000004" }
|
||||
|
||||
function makeDeps(enabled = true) {
|
||||
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES, files] })
|
||||
const deps: KnowledgeDeps & { now: () => number } = {
|
||||
data,
|
||||
files: { data, store: fakeObjectStore(), policy: DEFAULT_FILES_POLICY },
|
||||
flags: { enabled: async () => enabled },
|
||||
now: () => Date.parse("2026-11-09T19:00:00.000Z"),
|
||||
}
|
||||
return { data, deps }
|
||||
}
|
||||
|
||||
describe("automatic PDF indexing of a project", () => {
|
||||
it("is off until an administrator names a collection, and records who turned it on", async () => {
|
||||
const { data, deps } = makeDeps()
|
||||
await data.forScope(scope).of(collections).insert({ name: "contracts", createdBy: ADMIN })
|
||||
expect(await getAutoIndex(deps, scope)).toEqual({ collectionName: null })
|
||||
|
||||
expect(await setAutoIndex(deps, scope, "contracts")).toEqual({ collectionName: "contracts" })
|
||||
expect(await getAutoIndex(deps, scope)).toEqual({ collectionName: "contracts" })
|
||||
const [row] = (await data.forScope(scope).of(autoIndex).list({ limit: 10 })).items
|
||||
expect(row).toMatchObject({ collectionName: "contracts", enabledBy: ADMIN, enabledAt: "2026-11-09T19:00:00.000Z" })
|
||||
// Another project of the same organization is not opted in.
|
||||
expect(await getAutoIndex(deps, otherProject)).toEqual({ collectionName: null })
|
||||
})
|
||||
|
||||
it("keeps one row per project: naming another collection moves it, and off removes it", async () => {
|
||||
const { data, deps } = makeDeps()
|
||||
for (const name of ["contracts", "invoices"])
|
||||
await data.forScope(scope).of(collections).insert({ name, createdBy: ADMIN })
|
||||
await setAutoIndex(deps, scope, "contracts")
|
||||
await setAutoIndex(deps, scope, "invoices")
|
||||
expect(
|
||||
(await data.forScope(scope).of(autoIndex).list({ limit: 10 })).items.map((row) => row.collectionName),
|
||||
).toEqual(["invoices"])
|
||||
expect(await setAutoIndex(deps, scope, null)).toEqual({ collectionName: null })
|
||||
expect((await data.forScope(scope).of(autoIndex).list({ limit: 10 })).items).toEqual([])
|
||||
// Off when already off writes nothing and answers the same.
|
||||
expect(await setAutoIndex(deps, scope, null)).toEqual({ collectionName: null })
|
||||
})
|
||||
|
||||
it("answers another project's collection as not found, and refuses while knowledge is off", async () => {
|
||||
const { data, deps } = makeDeps()
|
||||
await data.forScope(otherProject).of(collections).insert({ name: "theirs", createdBy: ADMIN })
|
||||
await expect(setAutoIndex(deps, scope, "theirs")).rejects.toMatchObject({ status: 404 })
|
||||
|
||||
const off = makeDeps(false)
|
||||
await off.data.forScope(scope).of(collections).insert({ name: "contracts", createdBy: ADMIN })
|
||||
await expect(setAutoIndex(off.deps, scope, "contracts")).rejects.toMatchObject({ status: 403 })
|
||||
expect((await off.data.forScope(scope).of(autoIndex).list({ limit: 10 })).items).toEqual([])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,134 @@
|
||||
// SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0
|
||||
import {
|
||||
buildRegistry,
|
||||
Container,
|
||||
contribute, // kit:guides
|
||||
type ModuleManifest,
|
||||
type TenantScope,
|
||||
} from "@conusai-kit/core"
|
||||
import { createMemoryDataSource, DATA_SOURCE, entitiesOf, JOB_QUEUE, OBJECT_STORE } from "@conusai-kit/data"
|
||||
import * as guidesKit from "@conusai-kit/guides" // kit:guides
|
||||
import { projects, workspaces } from "@conusai-kit/tenancy"
|
||||
import { fakeObjectStore } from "@conusai-kit/testing"
|
||||
import { expect, it } from "vitest"
|
||||
import { custodyModule } from "../../custody.js"
|
||||
import { OPTIONAL_MODULE_IDS } from "../../inclusion.js"
|
||||
import { appModules } from "../../index.js"
|
||||
import { documents, KNOWLEDGE_ENTITIES, KNOWLEDGE_SCOPE, KNOWLEDGE_SEARCH, knowledgeModule } from "../index.js"
|
||||
|
||||
const actor = { userId: "01900000-0000-7000-0000-000000000009" }
|
||||
const orgId = "01900000-0000-7000-0000-000000000001"
|
||||
/**
|
||||
* A module as the app composes it without the opt-in guides module: its contributions to the guides extension
|
||||
* points go with it, whichever module made them (an app's own module may bring code guides too).
|
||||
*/
|
||||
function withoutGuideContributions(entry: ModuleManifest): ModuleManifest {
|
||||
return entry.extends
|
||||
? { ...entry, extends: entry.extends.filter((item) => !item.point.id.startsWith("guides.")) }
|
||||
: entry
|
||||
}
|
||||
|
||||
// kit:guides:begin
|
||||
it("drops guides contributions from every module, not only notes", () => {
|
||||
// An app's own module with a code guide: knowledge stands in for it.
|
||||
const guided: ModuleManifest = {
|
||||
...knowledgeModule,
|
||||
extends: [...(knowledgeModule.extends ?? []), contribute(guidesKit.guidesCode, [])],
|
||||
}
|
||||
expect(withoutGuideContributions(guided).extends?.filter((item) => item.point.id.startsWith("guides."))).toEqual([])
|
||||
})
|
||||
// kit:guides:end
|
||||
|
||||
it("off retains the original Knowledge rows/schema and executes only source cleanup", async () => {
|
||||
const base = appModules.filter(
|
||||
(entry) => !OPTIONAL_MODULE_IDS.some((id) => id === entry.id) && entry.id !== "assistant",
|
||||
)
|
||||
const independent = base.map(withoutGuideContributions)
|
||||
const active = buildRegistry([...independent, knowledgeModule])
|
||||
const retained = buildRegistry([...independent, custodyModule(knowledgeModule)])
|
||||
expect(retained.migrations.postgres).toEqual(active.migrations.postgres)
|
||||
expect(entitiesOf(retained, "postgres").map((entity) => entity.table)).toEqual(
|
||||
entitiesOf(active, "postgres").map((entity) => entity.table),
|
||||
)
|
||||
expect([...retained.procedures.keys()].filter((key) => key.startsWith("knowledge."))).toEqual([])
|
||||
expect(Object.keys(retained.jobs).filter((key) => key.startsWith("knowledge."))).toEqual(["knowledge.remove"])
|
||||
// The project the deleted file was in, as the organization has it: a delivery acts only in a project that exists.
|
||||
const data = createMemoryDataSource({ entities: [workspaces, projects, ...KNOWLEDGE_ENTITIES] })
|
||||
const workspace = await data
|
||||
.forScope({ level: "org", actor, orgId })
|
||||
.of(workspaces)
|
||||
.insert({ slug: "main", name: "Main", isDefault: true, visibility: "org" })
|
||||
const project = await data
|
||||
.forScope({ level: "workspace", actor, orgId, workspaceId: workspace.id })
|
||||
.of(projects)
|
||||
.insert({ slug: "site", name: "Site", moduleKey: "notes", archivedAt: null })
|
||||
const scope: TenantScope = { level: "project", actor, orgId, workspaceId: workspace.id, projectId: project.id }
|
||||
const row = await data.forScope(scope).of(documents).insert({
|
||||
collectionName: "synthetic",
|
||||
fileId: "01900000-0000-7000-0000-000000000004",
|
||||
filename: "synthetic.txt",
|
||||
status: "indexed",
|
||||
error: null,
|
||||
indexedAt: "2026-11-03T00:00:00Z",
|
||||
requestedBy: "synthetic",
|
||||
})
|
||||
const module = custodyModule(knowledgeModule)
|
||||
expect(await data.forScope(scope).of(documents).get(row.id)).toEqual(row)
|
||||
const container = new Container()
|
||||
container.value(DATA_SOURCE, data)
|
||||
container.value(OBJECT_STORE, fakeObjectStore())
|
||||
container.value(KNOWLEDGE_SCOPE, scope)
|
||||
const enqueued: string[] = []
|
||||
container.value(JOB_QUEUE, {
|
||||
enqueue: async (_scope, job) => {
|
||||
enqueued.push(job)
|
||||
return "synthetic-job"
|
||||
},
|
||||
})
|
||||
const deletion = module.events?.subscribes?.find((entry) => entry.event.name === "files.FileDeleted")
|
||||
if (!deletion) throw new Error("source cleanup subscription missing")
|
||||
await deletion.handler(
|
||||
{
|
||||
name: "files.FileDeleted",
|
||||
// As kit-files publishes it: the deleted file's storage key comes with the record.
|
||||
payload: {
|
||||
orgId: scope.orgId,
|
||||
workspaceId: scope.workspaceId,
|
||||
projectId: scope.projectId,
|
||||
fileId: row.fileId,
|
||||
storageKey: "synthetic/key",
|
||||
},
|
||||
},
|
||||
{ container } as never,
|
||||
)
|
||||
expect((await data.forScope(scope).of(documents).get(row.id))?.status).toBe("cancelled")
|
||||
const event = (await data.outbox.pending(10))[0]
|
||||
if (!event) throw new Error("cleanup event missing")
|
||||
const cleanup = module.events?.subscribes?.find((entry) => entry.event.name === "knowledge.documentRemoveRequested")
|
||||
if (!cleanup) throw new Error("cleanup subscription missing")
|
||||
await cleanup.handler({ name: event.name, payload: event.payload }, { container } as never)
|
||||
expect(enqueued).toEqual(["knowledge.remove"])
|
||||
expect(module.events?.subscribes?.some((entry) => entry.event.name === "knowledge.documentIndexRequested")).toBe(
|
||||
false,
|
||||
)
|
||||
const removed: string[] = []
|
||||
container.value(KNOWLEDGE_SEARCH, {
|
||||
remove: async (_scope: unknown, input: { fileId: string }) => {
|
||||
removed.push(input.fileId)
|
||||
},
|
||||
index: async () => {
|
||||
throw new Error("processing must be fenced")
|
||||
},
|
||||
search: async () => {
|
||||
throw new Error("interactive search must be fenced")
|
||||
},
|
||||
} as never)
|
||||
// The removal job resolves the original repositories and ports; no alternate store is created.
|
||||
const remove = module.jobs?.["knowledge.remove"]
|
||||
if (!remove) throw new Error("cleanup job missing")
|
||||
await remove.handler({ documentId: row.id, collectionName: row.collectionName, fileId: row.fileId }, {
|
||||
container,
|
||||
} as never)
|
||||
expect(removed).toEqual([row.fileId])
|
||||
expect(await data.forScope(scope).of(documents).get(row.id)).toBeNull()
|
||||
})
|
||||
@@ -0,0 +1,78 @@
|
||||
// SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0
|
||||
// A delivery's project comes from the record, inside the delivery's organization (adapters/jobs.ts). The record's
|
||||
// workspace and project must exist in that organization before knowledge acts on them; one that names another
|
||||
// workspace's project, or one the organization does not have, is refused by name and nothing is enqueued.
|
||||
|
||||
import { Container, type EventContext, newId, type OrgScope } from "@conusai-kit/core"
|
||||
import { createMemoryDataSource, DATA_SOURCE, JOB_QUEUE } from "@conusai-kit/data"
|
||||
import { projects, workspaces } from "@conusai-kit/tenancy"
|
||||
import { fakeJobQueue } from "@conusai-kit/testing"
|
||||
import { describe, expect, it } from "vitest"
|
||||
import { KNOWLEDGE_SUBSCRIPTIONS } from "../adapters/jobs.js"
|
||||
import { KNOWLEDGE_ENTITIES, KNOWLEDGE_SCOPE } from "../index.js"
|
||||
|
||||
async function delivery(archivedAt: string | null = null) {
|
||||
const data = createMemoryDataSource({ entities: [workspaces, projects, ...KNOWLEDGE_ENTITIES] })
|
||||
const orgId = newId()
|
||||
const org: OrgScope = { level: "org", actor: { userId: newId() }, orgId }
|
||||
const workspace = await data
|
||||
.forScope(org)
|
||||
.of(workspaces)
|
||||
.insert({ slug: "main", name: "Main", isDefault: true, visibility: "org" })
|
||||
const other = await data
|
||||
.forScope(org)
|
||||
.of(workspaces)
|
||||
.insert({ slug: "other", name: "Other", isDefault: false, visibility: "org" })
|
||||
const project = await data
|
||||
.forScope({ level: "workspace", actor: org.actor, orgId, workspaceId: workspace.id })
|
||||
.of(projects)
|
||||
.insert({ slug: "site", name: "Site", moduleKey: "notes", archivedAt })
|
||||
const queue = fakeJobQueue()
|
||||
const container = new Container()
|
||||
container.value(DATA_SOURCE, data)
|
||||
container.value(JOB_QUEUE, queue)
|
||||
// The relay's delivery: the organization's scope, with a system actor.
|
||||
container.value(KNOWLEDGE_SCOPE, org)
|
||||
const ctx = { container } as unknown as EventContext
|
||||
return { ctx, queue, orgId, workspaceId: workspace.id, otherWorkspaceId: other.id, projectId: project.id }
|
||||
}
|
||||
|
||||
const indexRequested = KNOWLEDGE_SUBSCRIPTIONS[1]
|
||||
const request = (ids: { orgId: string; workspaceId: string; projectId: string }) => ({
|
||||
name: "knowledge.DocumentIndexRequested",
|
||||
payload: { ...ids, documentId: newId(), collectionName: "contracts", fileId: newId() },
|
||||
})
|
||||
|
||||
describe("a knowledge delivery acts only in a project of its organization", () => {
|
||||
it("enqueues the job in the record's project when it exists there", async () => {
|
||||
const { ctx, queue, orgId, workspaceId, projectId } = await delivery()
|
||||
await indexRequested?.handler(request({ orgId, workspaceId, projectId }) as never, ctx)
|
||||
expect(queue.of("knowledge.index")).toHaveLength(1)
|
||||
})
|
||||
|
||||
it("refuses, by name, a workspace the organization does not have", async () => {
|
||||
const { ctx, queue, orgId, projectId } = await delivery()
|
||||
const workspaceId = newId()
|
||||
await expect(indexRequested?.handler(request({ orgId, workspaceId, projectId }) as never, ctx)).rejects.toThrow(
|
||||
new RegExp(`workspace ${workspaceId}`),
|
||||
)
|
||||
expect(queue.of("knowledge.index")).toEqual([])
|
||||
})
|
||||
|
||||
it("refuses, by name, a project that is not in the record's workspace", async () => {
|
||||
const { ctx, queue, orgId, otherWorkspaceId, projectId } = await delivery()
|
||||
await expect(
|
||||
indexRequested?.handler(request({ orgId, workspaceId: otherWorkspaceId, projectId }) as never, ctx),
|
||||
).rejects.toThrow(new RegExp(`project ${projectId}`))
|
||||
expect(queue.of("knowledge.index")).toEqual([])
|
||||
})
|
||||
|
||||
it("acts on a record of an archived project: an archive hides a project and does not end its work", async () => {
|
||||
// The kit's one rule (kit-tenancy suspension.ts): an archived project refuses a person's write, not work with
|
||||
// no person behind it. A deleted file's text must still leave the index, and an index request a member made
|
||||
// before the archive still completes.
|
||||
const { ctx, queue, orgId, workspaceId, projectId } = await delivery(new Date("2026-11-09T00:00:00Z").toISOString())
|
||||
await indexRequested?.handler(request({ orgId, workspaceId, projectId }) as never, ctx)
|
||||
expect(queue.of("knowledge.index")).toHaveLength(1)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,349 @@
|
||||
// The document requests and the index job's orchestration: idempotent (collection, file) pairs,
|
||||
// the flag gate, the content-type and size admission, the row's progress states, and which
|
||||
// failures are permanent.
|
||||
|
||||
import type { TenantScope } from "@conusai-kit/core"
|
||||
import { createMemoryDataSource } from "@conusai-kit/data"
|
||||
import { DEFAULT_FILES_POLICY, files, sha256Hex } from "@conusai-kit/files"
|
||||
import { fakeObjectStore } from "@conusai-kit/testing"
|
||||
import { describe, expect, it } from "vitest"
|
||||
import { collections, documents, KNOWLEDGE_ENTITIES } from "../contract/index.js"
|
||||
import { listDocuments, requestIndex, requestRemove, requestSourceRemoval } from "../core/documents.js"
|
||||
import { indexDocument, removeDocument } from "../core/index-job.js"
|
||||
import type { KnowledgeDeps } from "../core/ports.js"
|
||||
import { searchKnowledge } from "../core/search.js"
|
||||
|
||||
const USER = "01900000-0000-7000-0000-000000000009"
|
||||
const scope: TenantScope = {
|
||||
level: "project",
|
||||
actor: { userId: USER },
|
||||
orgId: "01900000-0000-7000-0000-000000000001",
|
||||
workspaceId: "01900000-0000-7000-0000-000000000002",
|
||||
projectId: "01900000-0000-7000-0000-000000000003",
|
||||
}
|
||||
|
||||
const fileRow = {
|
||||
status: "stored" as const,
|
||||
storageKey: "k/seed",
|
||||
filename: "a.txt",
|
||||
contentType: "text/plain",
|
||||
size: 12,
|
||||
sha256: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
||||
uploadedBy: USER,
|
||||
storedAt: "2026-10-23T00:00:00.000Z",
|
||||
uploadId: null,
|
||||
declaredSize: null,
|
||||
partSize: null,
|
||||
deletingAt: null,
|
||||
}
|
||||
|
||||
const makeDeps = (options: { flag?: boolean; search?: KnowledgeDeps["search"] } = {}) => {
|
||||
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES, files] })
|
||||
const store = fakeObjectStore()
|
||||
const search = options.search ?? {
|
||||
readiness: async () => ({ state: "ready" as const, reasons: [] }),
|
||||
index: async () => undefined,
|
||||
search: async () => [],
|
||||
remove: async () => undefined,
|
||||
}
|
||||
const deps: KnowledgeDeps = {
|
||||
data,
|
||||
files: { data, store, policy: DEFAULT_FILES_POLICY },
|
||||
search,
|
||||
flags: { enabled: async () => options.flag ?? true },
|
||||
}
|
||||
void deps
|
||||
return { data, deps, store }
|
||||
}
|
||||
|
||||
const seed = async (data: ReturnType<typeof createMemoryDataSource>, contentType = "text/plain") => {
|
||||
await data.forScope(scope).of(collections).insert({ name: "contracts", createdBy: USER })
|
||||
const row = await data
|
||||
.forScope(scope)
|
||||
.of(files)
|
||||
.insert({ ...fileRow, ...(contentType === "text/plain" ? {} : { contentType }) })
|
||||
return row.id
|
||||
}
|
||||
|
||||
/** Seeds the file row AND the object the row names, the way a completed upload leaves both. */
|
||||
const seedWithBytes = async (
|
||||
data: ReturnType<typeof createMemoryDataSource>,
|
||||
store: ReturnType<typeof fakeObjectStore>,
|
||||
bytes: Uint8Array,
|
||||
) => {
|
||||
const fileId = await seed(data)
|
||||
await data
|
||||
.forScope(scope)
|
||||
.of(files)
|
||||
.update(fileId, { size: bytes.length, sha256: await sha256Hex(bytes) }, 1)
|
||||
await store.put(scope, { key: fileRow.storageKey, body: bytes, contentType: "text/plain" })
|
||||
return fileId
|
||||
}
|
||||
|
||||
describe("requesting a document's indexing", () => {
|
||||
it("creates the row, publishes the event, and a repeat request changes nothing", async () => {
|
||||
const { data, deps } = makeDeps()
|
||||
const fileId = await seed(data)
|
||||
const first = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
|
||||
expect(first.status).toBe("queued")
|
||||
const second = await requestIndex(deps, scope, { collectionName: "contracts", fileId: first.fileId })
|
||||
expect(second.documentId).toBe(first.documentId)
|
||||
expect(await listDocuments(deps, scope, "contracts")).toEqual({
|
||||
documents: [expect.objectContaining({ documentId: first.documentId, status: "queued" })],
|
||||
})
|
||||
// One outbox event: the subscriber's enqueue is the exactly-once path.
|
||||
const outbox = data.outbox.pending(10)
|
||||
expect((await outbox).filter((record) => record.name === "knowledge.documentIndexRequested")).toHaveLength(1)
|
||||
})
|
||||
|
||||
it("re-queues an indexed pair instead of duplicating it", async () => {
|
||||
const { data, deps } = makeDeps()
|
||||
const fileId = await seed(data)
|
||||
const first = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
|
||||
await data.forScope(scope).of(documents).update(first.documentId, { status: "indexed" }, 1)
|
||||
const again = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
|
||||
expect(again.documentId).toBe(first.documentId)
|
||||
expect(again.status).toBe("queued")
|
||||
})
|
||||
|
||||
it("refuses when the flag is off, and admits text files only, within the size cap", async () => {
|
||||
const { data, deps } = makeDeps({ flag: false })
|
||||
const fileId = await seed(data)
|
||||
const off = await requestIndex(deps, scope, { collectionName: "contracts", fileId }).catch((e: unknown) => e)
|
||||
expect((off as { status?: number }).status).toBe(403)
|
||||
|
||||
const refusing = makeDeps()
|
||||
const pdfFile = await seed(refusing.data, "application/pdf")
|
||||
const pdf = await requestIndex(refusing.deps, scope, { collectionName: "contracts", fileId: pdfFile }).catch(
|
||||
(e: unknown) => e,
|
||||
)
|
||||
expect((pdf as { status?: number }).status).toBe(422)
|
||||
|
||||
const big = makeDeps()
|
||||
const bigFile = await seed(big.data)
|
||||
await big.data.forScope(scope).of(files).update(bigFile, { size: 2_000_000 }, 1)
|
||||
const tooBig = await requestIndex(big.deps, scope, { collectionName: "contracts", fileId: bigFile }).catch(
|
||||
(e: unknown) => e,
|
||||
)
|
||||
expect((tooBig as { status?: number }).status).toBe(422)
|
||||
})
|
||||
})
|
||||
|
||||
describe("current retrieval authority", () => {
|
||||
it("withholds cancelled and unindexed documents even while remote deletion retries", async () => {
|
||||
const { data, deps } = makeDeps()
|
||||
const fileId = await seed(data)
|
||||
const row = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
|
||||
const searching = {
|
||||
...deps,
|
||||
search: {
|
||||
...deps.search,
|
||||
readiness: async () => ({ state: "ready" as const, reasons: [] }),
|
||||
index: async () => undefined,
|
||||
remove: async () => undefined,
|
||||
search: async () => [
|
||||
{ fileId, filename: "a.txt", collectionName: "contracts", snippet: "private source", score: 1, chunk: 1 },
|
||||
],
|
||||
},
|
||||
}
|
||||
expect((await searchKnowledge(searching, scope, { collectionName: "contracts", query: "source" })).results).toEqual(
|
||||
[],
|
||||
)
|
||||
await data.forScope(scope).of(documents).update(row.documentId, { status: "indexed" }, 1)
|
||||
expect(
|
||||
(await searchKnowledge(searching, scope, { collectionName: "contracts", query: "source" })).results,
|
||||
).toHaveLength(1)
|
||||
await requestRemove(deps, scope, { collectionName: "contracts", fileId })
|
||||
expect((await searchKnowledge(searching, scope, { collectionName: "contracts", query: "source" })).results).toEqual(
|
||||
[],
|
||||
)
|
||||
// A removal a person asks for names the stored file's object, read when it is asked for: if the file is deleted
|
||||
// before the job runs, the job still removes the right text (F407, review R1).
|
||||
const asked = (await data.outbox.pending(100)).filter((event) => event.name === "knowledge.documentRemoveRequested")
|
||||
expect(asked).toHaveLength(1)
|
||||
expect(asked[0]?.payload).toMatchObject({ storageKey: "k/seed" })
|
||||
})
|
||||
|
||||
it("refuses a person's removal of a file being deleted, which leaves the document to the deletion's own removal", async () => {
|
||||
const { data, deps } = makeDeps()
|
||||
const fileId = await seed(data)
|
||||
const row = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
|
||||
await data.forScope(scope).of(documents).update(row.documentId, { status: "indexed" }, 1)
|
||||
const file = await data.forScope(scope).of(files).require(fileId)
|
||||
await data
|
||||
.forScope(scope)
|
||||
.of(files)
|
||||
.update(fileId, { status: "deleting", deletingAt: new Date().toISOString() }, file.version)
|
||||
await expect(requestRemove(deps, scope, { collectionName: "contracts", fileId })).rejects.toMatchObject({
|
||||
status: 409,
|
||||
})
|
||||
// Still active, so `FileDeleted` (which carries the key) cancels it and asks for the removal.
|
||||
expect((await data.forScope(scope).of(documents).require(row.documentId)).status).toBe("indexed")
|
||||
expect(
|
||||
(await data.outbox.pending(100)).filter((event) => event.name === "knowledge.documentRemoveRequested"),
|
||||
).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe("retained remote citations after source deletion", () => {
|
||||
it("withholds a deleting or missing file even when its index row still says indexed", async () => {
|
||||
const { data, deps } = makeDeps()
|
||||
const fileId = await seed(data)
|
||||
const row = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
|
||||
await data.forScope(scope).of(documents).update(row.documentId, { status: "indexed" }, 1)
|
||||
if (!deps.search) throw new Error("the retained citation fixture must configure search")
|
||||
const searching = {
|
||||
...deps,
|
||||
search: {
|
||||
...deps.search,
|
||||
search: async () => [
|
||||
{ fileId, filename: "a.txt", collectionName: "contracts", snippet: "retained source", score: 1, chunk: 1 },
|
||||
],
|
||||
},
|
||||
}
|
||||
expect(
|
||||
(await searchKnowledge(searching, scope, { collectionName: "contracts", query: "source" })).results,
|
||||
).toHaveLength(1)
|
||||
await data.forScope(scope).of(files).update(fileId, { status: "deleting" }, 1)
|
||||
expect((await searchKnowledge(searching, scope, { collectionName: "contracts", query: "source" })).results).toEqual(
|
||||
[],
|
||||
)
|
||||
const { storageKey } = (await data.forScope(scope).of(files).get(fileId)) ?? { storageKey: "" }
|
||||
await data.forScope(scope).of(files).delete(fileId)
|
||||
await requestSourceRemoval({ ...deps, flags: { enabled: () => false } }, scope, fileId, storageKey)
|
||||
await requestSourceRemoval(deps, scope, fileId, storageKey)
|
||||
expect((await listDocuments(deps, scope, "contracts")).documents[0]?.status).toBe("cancelled")
|
||||
const cleanup = (await data.outbox.pending(100)).filter(
|
||||
(event) => event.name === "knowledge.documentRemoveRequested",
|
||||
)
|
||||
expect(cleanup).toHaveLength(1)
|
||||
// The deleted file's key travels with the request: its row is gone, so the removal cannot look it up (F407).
|
||||
expect(cleanup[0]?.payload).toMatchObject({ storageKey })
|
||||
expect((await searchKnowledge(searching, scope, { collectionName: "contracts", query: "source" })).results).toEqual(
|
||||
[],
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
describe("the index job's orchestration", () => {
|
||||
it("moves the row queued -> indexing -> indexed and calls the search service once", async () => {
|
||||
const indexed: string[] = []
|
||||
const { data, deps, store } = makeDeps({
|
||||
search: {
|
||||
readiness: async () => ({ state: "ready", reasons: [] }),
|
||||
index: async (_scope, input) => {
|
||||
indexed.push(input.fileId)
|
||||
},
|
||||
search: async () => [],
|
||||
remove: async () => undefined,
|
||||
},
|
||||
})
|
||||
const fileId = await seedWithBytes(data, store, new TextEncoder().encode("hello knowledge"))
|
||||
const requested = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
|
||||
const done = await indexDocument(deps, scope, requested.documentId)
|
||||
expect(done.status).toBe("indexed")
|
||||
expect(indexed).toEqual([fileId])
|
||||
// A redelivered job finds the row indexed and does the external work again never.
|
||||
const again = await indexDocument(deps, scope, requested.documentId)
|
||||
expect(again.status).toBe("indexed")
|
||||
expect(indexed).toEqual([fileId])
|
||||
})
|
||||
|
||||
it("records a permanent failure on the row; a transient one throws for the queue to retry", async () => {
|
||||
const permanent = makeDeps({
|
||||
search: {
|
||||
readiness: async () => ({ state: "ready", reasons: [] }),
|
||||
index: async () => {
|
||||
throw Object.assign(new Error("not text"), { permanent: "not_text" })
|
||||
},
|
||||
search: async () => [],
|
||||
remove: async () => undefined,
|
||||
},
|
||||
})
|
||||
const permanentFile = await seedWithBytes(
|
||||
permanent.data,
|
||||
permanent.store,
|
||||
new TextEncoder().encode("not text at all"),
|
||||
)
|
||||
const requested = await requestIndex(permanent.deps, scope, { collectionName: "contracts", fileId: permanentFile })
|
||||
const failed = await indexDocument(permanent.deps, scope, requested.documentId)
|
||||
expect(failed.status).toBe("failed")
|
||||
expect(failed.error).toBe("not_text")
|
||||
|
||||
const transient = makeDeps({
|
||||
search: {
|
||||
readiness: async () => ({ state: "ready", reasons: [] }),
|
||||
index: async () => {
|
||||
throw new Error("the store hiccuped")
|
||||
},
|
||||
search: async () => [],
|
||||
remove: async () => undefined,
|
||||
},
|
||||
})
|
||||
const transientFile = await seedWithBytes(transient.data, transient.store, new TextEncoder().encode("hello"))
|
||||
const requested2 = await requestIndex(transient.deps, scope, { collectionName: "contracts", fileId: transientFile })
|
||||
await expect(indexDocument(transient.deps, scope, requested2.documentId)).rejects.toThrow("hiccuped")
|
||||
let resumed = 0
|
||||
const retry = {
|
||||
...transient.deps,
|
||||
search: {
|
||||
...transient.deps.search,
|
||||
readiness: async () => ({ state: "ready" as const, reasons: [] }),
|
||||
search: async () => [],
|
||||
remove: async () => undefined,
|
||||
index: async () => {
|
||||
resumed++
|
||||
},
|
||||
},
|
||||
}
|
||||
expect((await indexDocument(retry, scope, requested2.documentId)).status).toBe("indexed")
|
||||
expect(resumed).toBe(1)
|
||||
})
|
||||
|
||||
it("removal deletes the row and asks the service to drop the vectors", async () => {
|
||||
const removed: string[] = []
|
||||
const { data, deps } = makeDeps({
|
||||
search: {
|
||||
readiness: async () => ({ state: "ready", reasons: [] }),
|
||||
index: async () => undefined,
|
||||
search: async () => [],
|
||||
remove: async (_scope, input) => {
|
||||
removed.push(input.fileId)
|
||||
},
|
||||
},
|
||||
})
|
||||
const fileId = await seed(data)
|
||||
const requested = await requestIndex(deps, scope, { collectionName: "contracts", fileId })
|
||||
const cancelled = await requestRemove(deps, scope, { collectionName: "contracts", fileId })
|
||||
expect(cancelled.status).toBe("cancelled")
|
||||
expect((await indexDocument(deps, scope, requested.documentId)).status).toBe("cancelled")
|
||||
// The remove job's body: delete the vectors, then the row.
|
||||
await removeDocument(deps, scope, requested.documentId)
|
||||
expect(removed).toEqual([fileId])
|
||||
expect((await listDocuments(deps, scope, "contracts")).documents).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
it("persists a final transient indexing failure but never overwrites cancellation on a lost claim", async () => {
|
||||
const transient = makeDeps({
|
||||
search: {
|
||||
readiness: async () => ({ state: "ready", reasons: [] }),
|
||||
index: async () => {
|
||||
throw new Error("temporary provider outage")
|
||||
},
|
||||
search: async () => [],
|
||||
remove: async () => {},
|
||||
},
|
||||
})
|
||||
const fileId = await seedWithBytes(transient.data, transient.store, new TextEncoder().encode("synthetic source"))
|
||||
const requested = await requestIndex(transient.deps, scope, { collectionName: "contracts", fileId })
|
||||
const outcome = await indexDocument(transient.deps, scope, requested.documentId, { finalAttempt: true })
|
||||
expect(outcome).toMatchObject({ status: "failed", error: "processing_exhausted" })
|
||||
const controller = new AbortController()
|
||||
controller.abort()
|
||||
const next = await requestIndex(transient.deps, scope, { collectionName: "contracts", fileId })
|
||||
await expect(
|
||||
indexDocument(transient.deps, scope, next.documentId, { finalAttempt: true, signal: controller.signal }),
|
||||
).rejects.toBeDefined()
|
||||
expect((await transient.data.forScope(scope).of(documents).get(next.documentId))?.status).not.toBe("failed")
|
||||
})
|
||||
@@ -0,0 +1,30 @@
|
||||
// The module's messages: en and lt declare the same keys, or a page in one language breaks.
|
||||
|
||||
import { buildRegistry } from "@conusai-kit/core"
|
||||
import { filesModule } from "@conusai-kit/files"
|
||||
import { createI18n } from "@conusai-kit/i18n"
|
||||
import { tenancyModule } from "@conusai-kit/tenancy"
|
||||
import { describe, expect, it } from "vitest"
|
||||
import { knowledgeModule } from "../index.js"
|
||||
|
||||
describe("the knowledge module's messages", () => {
|
||||
const catalogues = buildRegistry([tenancyModule, filesModule, knowledgeModule]).catalogues
|
||||
|
||||
it("declares en and lt, and every en key resolves in lt", () => {
|
||||
const en = catalogues.en?.knowledge ?? {}
|
||||
const lt = catalogues.lt?.knowledge ?? {}
|
||||
const flatten = (tree: Record<string, unknown>, prefix = ""): string[] =>
|
||||
Object.entries(tree).flatMap(([key, value]) =>
|
||||
typeof value === "string" ? [`${prefix}${key}`] : flatten(value as Record<string, unknown>, `${prefix}${key}.`),
|
||||
)
|
||||
const enKeys = flatten(en)
|
||||
const ltKeys = new Set(flatten(lt))
|
||||
expect(enKeys.length).toBeGreaterThan(0)
|
||||
for (const key of enKeys) expect(ltKeys.has(key), `knowledge.${key} has no Lithuanian message`).toBe(true)
|
||||
})
|
||||
|
||||
it("formats a message with values in both locales", () => {
|
||||
const i18n = createI18n("lt", { lt: { knowledge: catalogues.lt?.knowledge ?? {} } })
|
||||
expect(i18n.message("knowledge.status.documents", { queued: 1, indexing: 2, indexed: 3, failed: 0 })).toContain("1")
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,59 @@
|
||||
// What the generated tenant-isolation suite (test/isolation.test.ts) needs of this module: rows it
|
||||
// seeds in every tenant, and one fixture per tenant-scoped procedure. A fixture is built from the
|
||||
// OTHER tenant's ids: the call is made as a member of organization A with organization B's slugs
|
||||
// and ids, and has to be refused or answered empty.
|
||||
|
||||
import type { TransportCaller } from "@conusai-kit/core"
|
||||
import { type IsolationFixture, type IsolationTenant, uploadViaProcedures } from "@conusai-kit/testing"
|
||||
|
||||
/** What a module's isolation file exports: the shape the suite reads. */
|
||||
export interface ModuleIsolation {
|
||||
seed(
|
||||
call: TransportCaller,
|
||||
inProject: { org: string; workspace: string; project: string },
|
||||
): Promise<Record<string, string>>
|
||||
fixtures: Record<string, IsolationFixture>
|
||||
}
|
||||
|
||||
const NOWHERE = "00000000-0000-0000-0000-000000000000"
|
||||
const inProjectOf = (t: IsolationTenant) => ({ org: t.orgSlug, workspace: t.workspaceSlug, project: t.projectSlug })
|
||||
const collectionOf = (t: IsolationTenant) => t.extra.knowledgeCollection ?? "contracts"
|
||||
const fileOf = (t: IsolationTenant) => t.extra.knowledgeFileId ?? NOWHERE
|
||||
/** A knowledge_document row the suite seeds in each tenant (test/isolation.test.ts): indexing needs the module on. */
|
||||
const documentOf = (t: IsolationTenant) => t.extra.knowledgeDocumentId ?? NOWHERE
|
||||
|
||||
export const isolation: ModuleIsolation = {
|
||||
async seed(call, inProject) {
|
||||
const file = await uploadViaProcedures(call, inProject, {
|
||||
filename: "knowledge-seed.txt",
|
||||
contentType: "text/plain",
|
||||
bytes: new TextEncoder().encode("seed knowledge"),
|
||||
})
|
||||
// The seed needs a collection row to exist; collection.create needs the admin role, which the
|
||||
// isolation harness does not grant the seeding caller, so the fixture names a collection and
|
||||
// the write fixtures create it through the manage permission where a denial is the assertion.
|
||||
return { knowledgeCollection: "contracts", knowledgeFileId: file.fileId }
|
||||
},
|
||||
fixtures: {
|
||||
"knowledge.collection.list": { input: (t) => inProjectOf(t) },
|
||||
"knowledge.collection.create": { input: (t) => ({ ...inProjectOf(t), name: "intruders" }) },
|
||||
"knowledge.document.list": {
|
||||
input: (t) => ({ ...inProjectOf(t), collectionName: collectionOf(t) }),
|
||||
},
|
||||
// The live collection's read-back: another tenant's document id answers nothing.
|
||||
"knowledge.document.getMany": { input: (t) => ({ ...inProjectOf(t), ids: [documentOf(t)] }) },
|
||||
"knowledge.document.index": {
|
||||
input: (t) => ({ ...inProjectOf(t), collectionName: collectionOf(t), fileId: fileOf(t) }),
|
||||
},
|
||||
"knowledge.document.remove": {
|
||||
input: (t) => ({ ...inProjectOf(t), collectionName: collectionOf(t), fileId: fileOf(t) }),
|
||||
},
|
||||
"knowledge.search": { input: (t) => ({ ...inProjectOf(t), collectionName: collectionOf(t), query: "seed" }) },
|
||||
"knowledge.status": { input: (t) => inProjectOf(t) },
|
||||
"knowledge.settings.get": { input: (t) => ({ org: t.orgSlug }) },
|
||||
"knowledge.settings.enableDocumentAi": { input: (t) => ({ org: t.orgSlug }) },
|
||||
"knowledge.settings.disableDocumentAi": { input: (t) => ({ org: t.orgSlug }) },
|
||||
"knowledge.autoindex.get": { input: (t) => inProjectOf(t) },
|
||||
"knowledge.autoindex.set": { input: (t) => ({ ...inProjectOf(t), collectionName: collectionOf(t) }) },
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
// The module's migrations are its own files, each exactly `schemaStatements()` over the entities it adds
|
||||
// (decision AT-D20: the entity definitions are the one source of a column). The module ships the
|
||||
// Postgres profile; the Convex binding is provided for the integration owner and has no SQL.
|
||||
|
||||
import { readFileSync } from "node:fs"
|
||||
import { schemaStatements } from "@conusai-kit/data-kysely"
|
||||
import { describe, expect, it } from "vitest"
|
||||
import { autoIndex, KNOWLEDGE_ENTITIES, KNOWLEDGE_ENTITIES_V1, knowledgeModule, passageSets } from "../index.js"
|
||||
|
||||
/** The statements for these tables alone: `schemaStatements` leads with the kernel's own two. */
|
||||
function tablesOf(entities: Parameters<typeof schemaStatements>[0]): string {
|
||||
const kernel = schemaStatements([]).length
|
||||
return `${schemaStatements(entities)
|
||||
.slice(kernel)
|
||||
.map((statement) => `${statement};`)
|
||||
.join("\n\n")}\n`
|
||||
}
|
||||
|
||||
describe("the migration of the knowledge module", () => {
|
||||
it("is its own file, exactly its three tables as first shipped", async () => {
|
||||
await expect(
|
||||
`-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0\n${tablesOf(KNOWLEDGE_ENTITIES_V1)}`,
|
||||
).toMatchFileSnapshot("../migrations/0001_knowledge.sql")
|
||||
})
|
||||
|
||||
it("adds the auto-index opt-in in the second file, exactly its table", async () => {
|
||||
await expect(
|
||||
`-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0\n${tablesOf([autoIndex])}`,
|
||||
).toMatchFileSnapshot("../migrations/0002_knowledge_auto_index.sql")
|
||||
})
|
||||
|
||||
it("adds the PDF intake's passage record in the third file, exactly its table", async () => {
|
||||
await expect(
|
||||
`-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0\n${tablesOf([passageSets])}`,
|
||||
).toMatchFileSnapshot("../migrations/0003_knowledge_passage_set.sql")
|
||||
expect(KNOWLEDGE_ENTITIES).toEqual([...KNOWLEDGE_ENTITIES_V1, autoIndex, passageSets])
|
||||
})
|
||||
|
||||
it("is declared by the manifest, so the migrate job finds it, and every table is row-level secured", () => {
|
||||
expect(knowledgeModule.migrations).toEqual({ postgres: "migrations" })
|
||||
expect(knowledgeModule.profiles).toEqual(["postgres"])
|
||||
const sql = ["0001_knowledge.sql", "0002_knowledge_auto_index.sql", "0003_knowledge_passage_set.sql"]
|
||||
.map((file) => readFileSync(new URL(`../migrations/${file}`, import.meta.url), "utf8"))
|
||||
.join("\n")
|
||||
for (const table of [
|
||||
"knowledge_collection",
|
||||
"knowledge_document",
|
||||
"knowledge_config",
|
||||
"knowledge_auto_index",
|
||||
"knowledge_passage_set",
|
||||
]) {
|
||||
expect(sql).toContain(`create table "${table}"`)
|
||||
expect(sql).toContain(`alter table "${table}" force row level security`)
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,208 @@
|
||||
// The knowledge page on F10: its loader reads through the caller and returns plain data with every label
|
||||
// translated, and its view renders what reached the browser. The service's health, the indexing indicator
|
||||
// and the search stay three separate things: "indexing" is not "down".
|
||||
|
||||
import { AppError, buildRegistry, Container, notFound, type TransportCaller } from "@conusai-kit/core"
|
||||
import { builtInCatalogues, createI18n, uiLabels } from "@conusai-kit/i18n"
|
||||
import { createDataClient, DataProvider } from "@conusai-kit/ui/data"
|
||||
import { UiLabelsProvider } from "@conusai-kit/ui/primitives"
|
||||
import { renderToStaticMarkup as renderMarkup } from "react-dom/server"
|
||||
import { describe, expect, it, vi } from "vitest"
|
||||
import { appModules } from "../../index.js"
|
||||
import { modulePages } from "../../pages.js"
|
||||
import { knowledgePage } from "../ui/pages.js"
|
||||
|
||||
const kitStrings = uiLabels(createI18n("en", builtInCatalogues))
|
||||
const renderToStaticMarkup = (node: React.ReactNode) =>
|
||||
renderMarkup(<UiLabelsProvider labels={kitStrings}>{node}</UiLabelsProvider>)
|
||||
|
||||
const catalogues = buildRegistry([...appModules]).catalogues
|
||||
const i18n = createI18n("en", {
|
||||
en: { knowledge: catalogues.en?.knowledge ?? {}, app: { failed: "That did not work: {detail}" } },
|
||||
})
|
||||
const slugs = { org: "acme", workspace: "ws", project: "p" }
|
||||
|
||||
const callerOf = (answers: Record<string, unknown>, asked: unknown[] = []): TransportCaller =>
|
||||
({
|
||||
call: async (contract: { name: string }, input: unknown) => {
|
||||
asked.push([contract.name, input])
|
||||
const answer = answers[contract.name]
|
||||
if (answer instanceof Error) throw answer
|
||||
return answer
|
||||
},
|
||||
}) as unknown as TransportCaller
|
||||
|
||||
const load = (segments: string[], answers: Record<string, unknown>, asked?: unknown[]) =>
|
||||
knowledgePage.load({
|
||||
slugs,
|
||||
segments,
|
||||
search: {},
|
||||
caller: callerOf(answers, asked),
|
||||
i18n,
|
||||
container: new Container(),
|
||||
})
|
||||
|
||||
/** The page as the shell renders it: the loader's data through JSON, as the browser receives it, then the view. */
|
||||
const render = async (answers: Record<string, unknown>) => {
|
||||
const data = await load([], answers)
|
||||
expect(JSON.parse(JSON.stringify(data))).toStrictEqual(data)
|
||||
const { View } = knowledgePage
|
||||
return renderToStaticMarkup(
|
||||
<DataProvider client={createDataClient({ fetch: async () => new Response(null, { status: 501 }) })}>
|
||||
<View slugs={slugs} segments={[]} locale="en" data={JSON.parse(JSON.stringify(data))} />
|
||||
</DataProvider>,
|
||||
)
|
||||
}
|
||||
|
||||
const document = (documentId: string, status: string, error: string | null) => ({
|
||||
documentId,
|
||||
collectionName: "contracts",
|
||||
fileId: `file-${documentId}`,
|
||||
filename: `${documentId}.pdf`,
|
||||
status,
|
||||
error,
|
||||
indexedAt: null,
|
||||
requestedBy: "u1",
|
||||
})
|
||||
|
||||
const status = (enabled: boolean, state: string, reasons: readonly string[] = [], refusal?: string) => ({
|
||||
enabled,
|
||||
service: { state, reasons, ...(refusal === undefined ? {} : { refusal }) },
|
||||
documents: { queued: 1, indexing: 0, indexed: 2, failed: 0, cancelled: 0 },
|
||||
})
|
||||
|
||||
const answers = {
|
||||
"knowledge.collection.list": {
|
||||
collections: [{ name: "contracts", documentCount: 2, indexedCount: 1, createdBy: "u1" }],
|
||||
},
|
||||
"knowledge.status": status(true, "degraded", ["the search index is behind"]),
|
||||
"knowledge.document.list": {
|
||||
documents: [document("d1", "indexed", null), document("d2", "failed", "the file could not be read")],
|
||||
},
|
||||
"knowledge.autoindex.get": { collectionName: null },
|
||||
}
|
||||
|
||||
describe("the knowledge page", () => {
|
||||
it("is listed for the shell", async () => {
|
||||
await expect(modulePages.knowledge?.()).resolves.toBe(knowledgePage)
|
||||
})
|
||||
|
||||
it("shows the service's health, the collection and its documents in words", async () => {
|
||||
const html = await render(answers)
|
||||
expect(html).toContain("<h1 ")
|
||||
expect(html).toContain("<li>the search index is behind</li>")
|
||||
expect(html).toContain(">contracts</h2>")
|
||||
expect(html).toContain("d1.pdf")
|
||||
expect(html).toContain("the file could not be read")
|
||||
expect(html).toContain('role="grid"')
|
||||
expect(html.match(/role="columnheader"/g)).toHaveLength(3)
|
||||
})
|
||||
|
||||
// core-11: the project's automatic PDF indexing, off or naming its collection, with the project's slugs.
|
||||
it("offers automatic PDF indexing into one of the collections, showing the current choice", async () => {
|
||||
const asked: unknown[] = []
|
||||
const off = await load([], answers, asked)
|
||||
expect(asked).toContainEqual(["knowledge.autoindex.get", slugs])
|
||||
expect(off.autoIndex.current).toBeNull()
|
||||
const html = await render(answers)
|
||||
expect(html).toContain("Index uploaded PDFs automatically")
|
||||
const on = await load([], { ...answers, "knowledge.autoindex.get": { collectionName: "contracts" } })
|
||||
expect(on.autoIndex.current).toBe("contracts")
|
||||
})
|
||||
|
||||
it("does not offer automatic indexing while the module is off", async () => {
|
||||
const html = await render({ ...answers, "knowledge.status": status(false, "disabled") })
|
||||
expect(html).not.toContain("Index uploaded PDFs automatically")
|
||||
})
|
||||
|
||||
it("reads each collection's documents with the project's slugs", async () => {
|
||||
const asked: unknown[] = []
|
||||
await load([], answers, asked)
|
||||
expect(asked).toContainEqual(["knowledge.document.list", { ...slugs, collectionName: "contracts" }])
|
||||
})
|
||||
|
||||
it("says the module is off in place of its collections", async () => {
|
||||
const html = await render({ ...answers, "knowledge.status": status(false, "disabled") })
|
||||
expect(html).toContain('role="status"')
|
||||
expect(html).not.toContain('role="grid"')
|
||||
expect(html).not.toContain("<dl>")
|
||||
})
|
||||
|
||||
it.each([
|
||||
["plan_required", "The organization's plan does not include document search."],
|
||||
["feature_disabled", "Document search is switched off for this organization by platform policy."],
|
||||
["quota_exhausted", "Today's indexing allowance is spent. Try again tomorrow."],
|
||||
["budget_exhausted", "The organization's AI budget is spent. An administrator can raise it on the platform."],
|
||||
[
|
||||
"scope_missing",
|
||||
"This deployment's token does not hold knowledge.collections:use. Deploy the app again, then have an owner or admin approve it.",
|
||||
],
|
||||
["indexing", "The service is building its index. Try again in a few minutes."],
|
||||
["approval_required", "An administrator must approve this on the platform first."],
|
||||
["runtime_unavailable", "The knowledge service did not answer. Try again later."],
|
||||
])("shows the refusal %s as the page's own state", async (refusal, text) => {
|
||||
const html = await render({
|
||||
...answers,
|
||||
"knowledge.status": status(true, "unavailable", [], refusal),
|
||||
})
|
||||
expect(html).toContain(`data-state="${refusal}"`)
|
||||
expect(html).toContain('role="status"')
|
||||
expect(html).toContain(text.replaceAll("'", "'"))
|
||||
})
|
||||
|
||||
it("shows a refusal in Lithuanian to a Lithuanian reader", async () => {
|
||||
const lt = createI18n("lt", {
|
||||
lt: { knowledge: catalogues.lt?.knowledge ?? {}, app: { failed: "Nepavyko: {detail}" } },
|
||||
})
|
||||
const data = await knowledgePage.load({
|
||||
slugs,
|
||||
segments: [],
|
||||
search: {},
|
||||
caller: callerOf({ ...answers, "knowledge.status": status(true, "disabled", [], "plan_required") }),
|
||||
i18n: lt,
|
||||
container: new Container(),
|
||||
})
|
||||
expect(data.status.refusal).toEqual({
|
||||
code: "plan_required",
|
||||
text: "Organizacijos planas dokumentų paieškos neapima.",
|
||||
})
|
||||
})
|
||||
|
||||
it("says a document's failure in words, and keeps a code it does not know", async () => {
|
||||
const html = await render({
|
||||
...answers,
|
||||
"knowledge.document.list": {
|
||||
documents: [document("d1", "failed", "quota_exhausted"), document("d2", "failed", "something_new")],
|
||||
},
|
||||
})
|
||||
// Worded in the browser from the failure labels the loader handed over: the list is live.
|
||||
expect(html).toContain(">Today's indexing allowance is spent. Try again tomorrow.<")
|
||||
expect(html).toContain(">something_new<")
|
||||
})
|
||||
|
||||
it("answers a path it does not know as not found, without reading", async () => {
|
||||
const call = vi.fn()
|
||||
await expect(
|
||||
knowledgePage.load({
|
||||
slugs,
|
||||
segments: ["extra"],
|
||||
search: {},
|
||||
caller: { call } as unknown as TransportCaller,
|
||||
i18n,
|
||||
container: new Container(),
|
||||
}),
|
||||
).rejects.toMatchObject({ status: 404 })
|
||||
expect(call).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it("hands a procedure's refusal on, as the AppError the shell answers", async () => {
|
||||
const refused = load([], { ...answers, "knowledge.collection.list": notFound("no such project") })
|
||||
await expect(refused).rejects.toBeInstanceOf(AppError)
|
||||
await expect(refused).rejects.toMatchObject({ status: 404 })
|
||||
})
|
||||
})
|
||||
|
||||
// Explicit synthetic admission: module-local fixtures exercise the selected module.
|
||||
vi.mock("../../inclusion.json", () => ({
|
||||
default: { schemaVersion: 1, selected: ["knowledge"], everAdopted: ["knowledge"] },
|
||||
}))
|
||||
@@ -0,0 +1,304 @@
|
||||
// The PDF intake's two adapters (template lesson core-11). The reader hands a PDF to the platform's
|
||||
// document reader and answers its passages (`@conusai-sdk/ocr/passages`); the passage index sends one
|
||||
// inline item per passage, under the file's Blob reference plus `#p{page}.{index}` and with its page,
|
||||
// blocks and file in metadata, and tombstones exactly those references. A search hit on a passage
|
||||
// cites back into the file it came from.
|
||||
import { createMemoryDataSource } from "@conusai-kit/data"
|
||||
import { DEFAULT_FILES_POLICY, files } from "@conusai-kit/files"
|
||||
import {
|
||||
KNOWLEDGE_LIMITS,
|
||||
type Knowledge,
|
||||
type KnowledgeClient,
|
||||
KnowledgeError,
|
||||
type KnowledgeInlineItem,
|
||||
} from "@conusai-kit/platform"
|
||||
import { fakeObjectStore } from "@conusai-kit/testing"
|
||||
import { OcrError } from "@conusai-sdk/ocr"
|
||||
import { describe, expect, it } from "vitest"
|
||||
import { platformKnowledgeSearch, platformPassageIndex } from "../adapters/platform-knowledge.js"
|
||||
import { sdkKnowledgeReader } from "../adapters/reader.js"
|
||||
import type { KnowledgePassage, KnowledgeScope } from "../core/ports.js"
|
||||
|
||||
const scope: KnowledgeScope = {
|
||||
level: "project",
|
||||
actor: { userId: "system:knowledge.autoindex", system: true },
|
||||
orgId: "01900000-0000-7000-0000-000000000001",
|
||||
workspaceId: "01900000-0000-7000-0000-000000000002",
|
||||
projectId: "01900000-0000-7000-0000-000000000003",
|
||||
}
|
||||
const storageKey = `files/${scope.workspaceId}/${scope.projectId}/01900000-0000-7000-0000-000000000100`
|
||||
const pathname = `org/${scope.orgId}/${storageKey}`
|
||||
|
||||
async function storedFile() {
|
||||
const data = createMemoryDataSource({ entities: [files] })
|
||||
const deps = { data, store: fakeObjectStore(), policy: DEFAULT_FILES_POLICY }
|
||||
const row = await data
|
||||
.forScope(scope)
|
||||
.of(files)
|
||||
.insert({
|
||||
status: "stored",
|
||||
storageKey,
|
||||
filename: "contract.pdf",
|
||||
contentType: "application/pdf",
|
||||
size: 10,
|
||||
sha256: "0".repeat(64),
|
||||
uploadedBy: "01900000-0000-7000-0000-000000000010",
|
||||
storedAt: "2026-11-09T19:00:00.000Z",
|
||||
uploadId: null,
|
||||
declaredSize: null,
|
||||
partSize: null,
|
||||
deletingAt: null,
|
||||
})
|
||||
return { deps, fileId: row.id }
|
||||
}
|
||||
|
||||
type PassageClient = Pick<KnowledgeClient, "createCollection" | "indexInline" | "job" | "remove">
|
||||
|
||||
function passageClient(jobError: string | null = null) {
|
||||
const calls = {
|
||||
collections: [] as string[],
|
||||
batches: [] as KnowledgeInlineItem[][],
|
||||
polled: [] as string[],
|
||||
removed: [] as string[],
|
||||
}
|
||||
const client: PassageClient = {
|
||||
createCollection: async (input) => {
|
||||
calls.collections.push(`${input.collection}@${input.environment}`)
|
||||
return input
|
||||
},
|
||||
indexInline: async (input) => {
|
||||
calls.batches.push([...input.items])
|
||||
return input.items.map((item, at) => ({
|
||||
documentRef: item.documentRef,
|
||||
jobId: `job-${calls.batches.length}-${at}`,
|
||||
}))
|
||||
},
|
||||
job: async (jobId) => {
|
||||
calls.polled.push(jobId)
|
||||
return jobError === null
|
||||
? { jobId, status: "succeeded", progress: 1, generation: 1, error: null }
|
||||
: { jobId, status: "failed", progress: null, generation: null, error: jobError }
|
||||
},
|
||||
remove: async (documentRef) => {
|
||||
calls.removed.push(documentRef)
|
||||
},
|
||||
}
|
||||
return { client, calls }
|
||||
}
|
||||
|
||||
const passage = (page: number, index: number, blockIds: string[] = []): KnowledgePassage => ({
|
||||
page,
|
||||
index,
|
||||
blockIds,
|
||||
text: `page ${page}, passage ${index}`,
|
||||
})
|
||||
|
||||
describe("the passage index", () => {
|
||||
it("indexes one tenant item per passage under the file's reference, with its page, blocks and file in metadata", async () => {
|
||||
const { deps, fileId } = await storedFile()
|
||||
const { client, calls } = passageClient()
|
||||
const index = platformPassageIndex(client, deps, { environment: "production", intervalMs: 1 })
|
||||
|
||||
const documentRef = await index.reference(scope, fileId)
|
||||
expect(documentRef).toBe(pathname)
|
||||
await index.index(scope, {
|
||||
collectionName: "contracts",
|
||||
documentRef,
|
||||
fileId,
|
||||
passages: [passage(1, 0, ["b1", "b2"]), passage(2, 0)],
|
||||
})
|
||||
|
||||
expect(calls.collections).toEqual(["contracts@production"])
|
||||
const [batch] = calls.batches
|
||||
expect(batch?.map((item) => item.documentRef)).toEqual([`${pathname}#p1.0`, `${pathname}#p2.0`])
|
||||
expect(batch?.[0]).toMatchObject({
|
||||
text: "page 1, passage 0",
|
||||
visibility: "tenant",
|
||||
metadata: { page: "1", block_ids: "b1,b2", file_id: fileId },
|
||||
})
|
||||
// A passage on a page with no blocks carries no block ids rather than an empty one.
|
||||
expect(batch?.[1]?.metadata).toEqual({ page: "2", file_id: fileId })
|
||||
// The tenant partition is the project's, the same one a search of the project asks for.
|
||||
expect(new Set(batch?.map((item) => item.tenant))).toEqual(new Set([batch?.[0]?.tenant]))
|
||||
expect(batch?.[0]?.tenant).toMatch(/^[0-9a-f]{64}$/)
|
||||
// Every item's durable job is waited for.
|
||||
expect([...calls.polled].sort()).toEqual(["job-1-0", "job-1-1"])
|
||||
})
|
||||
|
||||
it("sends at most the service's item count per call", async () => {
|
||||
const { deps, fileId } = await storedFile()
|
||||
const { client, calls } = passageClient()
|
||||
const many = Array.from({ length: KNOWLEDGE_LIMITS.inlineItems + 1 }, (_, at) => passage(1, at))
|
||||
await platformPassageIndex(client, deps, { environment: "production", intervalMs: 1 }).index(scope, {
|
||||
collectionName: "contracts",
|
||||
documentRef: pathname,
|
||||
fileId,
|
||||
passages: many,
|
||||
})
|
||||
expect(calls.batches.map((batch) => batch.length)).toEqual([KNOWLEDGE_LIMITS.inlineItems, 1])
|
||||
})
|
||||
|
||||
it("records a quota refusal as the document's, not as something to retry", async () => {
|
||||
const { deps, fileId } = await storedFile()
|
||||
const { client } = passageClient("quota-exhausted")
|
||||
await expect(
|
||||
platformPassageIndex(client, deps, { environment: "production", intervalMs: 1 }).index(scope, {
|
||||
collectionName: "contracts",
|
||||
documentRef: pathname,
|
||||
fileId,
|
||||
passages: [passage(1, 0)],
|
||||
}),
|
||||
).rejects.toMatchObject({ reason: "quota_exhausted", permanent: "quota_exhausted" })
|
||||
})
|
||||
|
||||
it("tombstones exactly the references it indexed, without the file", async () => {
|
||||
const { deps } = await storedFile()
|
||||
const { client, calls } = passageClient()
|
||||
await platformPassageIndex(client, deps, { environment: "production" }).remove(scope, {
|
||||
documentRef: pathname,
|
||||
passages: [
|
||||
{ page: 1, index: 0 },
|
||||
{ page: 1, index: 1 },
|
||||
{ page: 3, index: 0 },
|
||||
],
|
||||
})
|
||||
expect(calls.removed).toEqual([`${pathname}#p1.0`, `${pathname}#p1.1`, `${pathname}#p3.0`])
|
||||
})
|
||||
|
||||
it("takes a passage the service never indexed as already removed", async () => {
|
||||
const { deps } = await storedFile()
|
||||
const { client, calls } = passageClient()
|
||||
const index = platformPassageIndex(
|
||||
{
|
||||
...client,
|
||||
remove: async (documentRef) => {
|
||||
calls.removed.push(documentRef)
|
||||
if (documentRef.endsWith("#p1.1")) throw new KnowledgeError("unavailable_resource", "no such document")
|
||||
},
|
||||
},
|
||||
deps,
|
||||
{ environment: "production" },
|
||||
)
|
||||
await index.remove(scope, {
|
||||
documentRef: pathname,
|
||||
passages: [
|
||||
{ page: 1, index: 0 },
|
||||
{ page: 1, index: 1 },
|
||||
{ page: 2, index: 0 },
|
||||
],
|
||||
})
|
||||
expect(calls.removed).toEqual([`${pathname}#p1.0`, `${pathname}#p1.1`, `${pathname}#p2.0`])
|
||||
})
|
||||
})
|
||||
|
||||
describe("a search hit on a passage", () => {
|
||||
it("cites the file it came from, with the passage's page and first block", async () => {
|
||||
const { deps, fileId } = await storedFile()
|
||||
const client = {
|
||||
search: async () => [
|
||||
{
|
||||
documentRef: `${pathname}#p2.0`,
|
||||
snippet: "page 2",
|
||||
score: 0.9,
|
||||
citations: [],
|
||||
chunkId: "chunk-1",
|
||||
metadata: { page: "2", block_ids: "b3,b4", file_id: fileId },
|
||||
blockIds: [],
|
||||
},
|
||||
],
|
||||
} as unknown as Knowledge
|
||||
const hits = await platformKnowledgeSearch(client, deps, { environment: "production" }).search(scope, {
|
||||
collectionName: "contracts",
|
||||
query: "page",
|
||||
limit: 5,
|
||||
actorId: "01900000-0000-7000-0000-000000000010",
|
||||
})
|
||||
expect(hits).toEqual([
|
||||
{
|
||||
fileId,
|
||||
filename: "contract.pdf",
|
||||
collectionName: "contracts",
|
||||
score: 0.9,
|
||||
chunk: 1,
|
||||
snippet: "page 2",
|
||||
page: 2,
|
||||
blockId: "b3",
|
||||
chunkId: "chunk-1",
|
||||
},
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
describe("the reader", () => {
|
||||
const document = {
|
||||
schema_version: "conusai.ocr.document/v1",
|
||||
pages: [
|
||||
{
|
||||
index: 0,
|
||||
lines: [
|
||||
{ text: "Agreement", order: 0, repeated: false },
|
||||
{ text: "Signed", order: 1, repeated: false },
|
||||
],
|
||||
blocks: [{ id: "p1-b1", order: 0, lines: [0, 1] }],
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
it("submits the PDF once under its key and answers the read document's passages", async () => {
|
||||
const submitted: Array<{ filename?: string; idempotencyKey?: string }> = []
|
||||
const reader = sdkKnowledgeReader(
|
||||
{
|
||||
submit: (async (_bytes: unknown, options: { filename?: string; idempotencyKey?: string }) => {
|
||||
submitted.push(options)
|
||||
return { id: 7 }
|
||||
}) as never,
|
||||
wait: (async (job: unknown) => {
|
||||
expect(job).toBe(7)
|
||||
return document
|
||||
}) as never,
|
||||
},
|
||||
{ intervalMs: 1 },
|
||||
)
|
||||
const read = await reader.read(new Uint8Array([1]), { filename: "contract.pdf", idempotencyKey: "key-1" })
|
||||
expect(submitted).toEqual([{ filename: "contract.pdf", idempotencyKey: "key-1" }])
|
||||
expect(read).toEqual([{ page: 1, index: 0, blockIds: ["p1-b1"], text: "Agreement\nSigned" }])
|
||||
})
|
||||
|
||||
it("answers a job the platform failed, or a document it cannot cite, as not indexable", async () => {
|
||||
const failing = sdkKnowledgeReader({
|
||||
submit: (async () => ({ id: 1 })) as never,
|
||||
wait: (async () => {
|
||||
throw new OcrError("failed", { type: "job-failed", status: 200 })
|
||||
}) as never,
|
||||
})
|
||||
await expect(failing.read(new Uint8Array([1]), { filename: "a.pdf", idempotencyKey: "k" })).rejects.toMatchObject({
|
||||
permanent: "not_indexable",
|
||||
})
|
||||
|
||||
const longId = {
|
||||
...document,
|
||||
pages: [{ ...document.pages[0], blocks: [{ id: "x".repeat(257), order: 0, lines: [0, 1] }] }],
|
||||
}
|
||||
const uncitable = sdkKnowledgeReader({
|
||||
submit: (async () => ({ id: 1 })) as never,
|
||||
wait: (async () => longId) as never,
|
||||
})
|
||||
await expect(uncitable.read(new Uint8Array([1]), { filename: "a.pdf", idempotencyKey: "k" })).rejects.toMatchObject(
|
||||
{
|
||||
permanent: "not_indexable",
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
it("leaves a transient failure to the queue", async () => {
|
||||
const transient = sdkKnowledgeReader({
|
||||
submit: (async () => {
|
||||
throw new OcrError("unavailable", { status: 503 })
|
||||
}) as never,
|
||||
wait: (async () => document) as never,
|
||||
})
|
||||
const error = await transient.read(new Uint8Array([1]), { filename: "a.pdf", idempotencyKey: "k" }).catch((e) => e)
|
||||
expect(error).toBeInstanceOf(OcrError)
|
||||
expect((error as { permanent?: unknown }).permanent).toBeUndefined()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,325 @@
|
||||
import { createMemoryDataSource } from "@conusai-kit/data"
|
||||
import { DEFAULT_FILES_POLICY, type FilesDeps, files } from "@conusai-kit/files"
|
||||
import { type Knowledge, KnowledgeError, type KnowledgeSearchHit } from "@conusai-kit/platform"
|
||||
import { fakeObjectStore } from "@conusai-kit/testing"
|
||||
import { describe, expect, it } from "vitest"
|
||||
import { platformKnowledgeSearch } from "../adapters/platform-knowledge.js"
|
||||
import type { KnowledgeScope } from "../core/ports.js"
|
||||
|
||||
const scope: KnowledgeScope = {
|
||||
level: "project",
|
||||
actor: { userId: "01900000-0000-7000-0000-000000000009" },
|
||||
orgId: "01900000-0000-7000-0000-000000000001",
|
||||
workspaceId: "01900000-0000-7000-0000-000000000002",
|
||||
projectId: "01900000-0000-7000-0000-000000000003",
|
||||
}
|
||||
const otherScope: KnowledgeScope = {
|
||||
...scope,
|
||||
projectId: "01900000-0000-7000-0000-000000000004",
|
||||
}
|
||||
const storageKey = `files/${scope.workspaceId}/${scope.projectId}/01900000-0000-7000-0000-000000000100`
|
||||
const otherStorageKey = `files/${otherScope.workspaceId}/${otherScope.projectId}/01900000-0000-7000-0000-000000000101`
|
||||
const pathname = `org/${scope.orgId}/${storageKey}`
|
||||
|
||||
interface ClientStub {
|
||||
indexed: string[]
|
||||
removed: string[]
|
||||
searches: Array<Parameters<Knowledge["search"]>[0]>
|
||||
results: readonly KnowledgeSearchHit[]
|
||||
}
|
||||
|
||||
function knowledgeClient(stub: ClientStub): Knowledge {
|
||||
return {
|
||||
readiness: async () => ({ state: "ready", reasons: [], checkedAt: new Date().toISOString() }),
|
||||
createCollection: async (input) => input,
|
||||
search: async (input) => {
|
||||
stub.searches.push(input)
|
||||
return stub.results
|
||||
},
|
||||
index: async (input) => {
|
||||
stub.indexed.push(input.documentRef)
|
||||
return { jobId: "job-1" }
|
||||
},
|
||||
job: async (jobId) => ({ jobId, status: "succeeded", progress: 1, generation: 1, error: null }),
|
||||
cancel: async () => undefined,
|
||||
remove: async (documentRef) => {
|
||||
stub.removed.push(documentRef)
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function stub(results: readonly KnowledgeSearchHit[] = []): ClientStub {
|
||||
return { indexed: [], removed: [], searches: [], results }
|
||||
}
|
||||
|
||||
function makeFiles() {
|
||||
const data = createMemoryDataSource({ entities: [files] })
|
||||
const deps: FilesDeps = { data, store: fakeObjectStore(), policy: DEFAULT_FILES_POLICY }
|
||||
return { data, deps }
|
||||
}
|
||||
|
||||
async function seedFile(
|
||||
data: ReturnType<typeof createMemoryDataSource>,
|
||||
atScope: KnowledgeScope,
|
||||
key: string,
|
||||
filename = "contract.txt",
|
||||
): Promise<string> {
|
||||
const row = await data.forScope(atScope).of(files).insert({
|
||||
status: "stored",
|
||||
storageKey: key,
|
||||
filename,
|
||||
contentType: "text/plain",
|
||||
size: 17,
|
||||
sha256: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
||||
uploadedBy: atScope.actor.userId,
|
||||
storedAt: "2026-10-24T00:00:00.000Z",
|
||||
uploadId: null,
|
||||
declaredSize: null,
|
||||
partSize: null,
|
||||
deletingAt: null,
|
||||
})
|
||||
return row.id
|
||||
}
|
||||
|
||||
describe("the managed knowledge Blob reference", () => {
|
||||
it("indexes and removes by Blob pathname rather than opaque file id", async () => {
|
||||
const { data, deps } = makeFiles()
|
||||
const fileId = await seedFile(data, scope, storageKey)
|
||||
const client = stub()
|
||||
const service = platformKnowledgeSearch(knowledgeClient(client), deps, { environment: "preview" })
|
||||
|
||||
await service.index(scope, {
|
||||
collectionName: "contracts",
|
||||
fileId,
|
||||
filename: "contract.txt",
|
||||
bytes: new TextEncoder().encode("managed knowledge"),
|
||||
})
|
||||
await service.remove(scope, { collectionName: "contracts", fileId })
|
||||
|
||||
expect(client.indexed).toEqual([pathname])
|
||||
expect(client.removed).toEqual([pathname])
|
||||
})
|
||||
|
||||
it("removes a deleted source by the key it had, whose files row is gone (F407)", async () => {
|
||||
const { data, deps } = makeFiles()
|
||||
const fileId = await seedFile(data, scope, storageKey)
|
||||
await data.forScope(scope).of(files).delete(fileId)
|
||||
const client = stub()
|
||||
const service = platformKnowledgeSearch(knowledgeClient(client), deps, { environment: "preview" })
|
||||
|
||||
// Without the key the file can no longer be found, so the removal refuses rather than guessing.
|
||||
await expect(service.remove(scope, { collectionName: "contracts", fileId })).rejects.toThrow()
|
||||
await service.remove(scope, { collectionName: "contracts", fileId, storageKey })
|
||||
|
||||
expect(client.removed).toEqual([pathname])
|
||||
})
|
||||
|
||||
it("maps citations through scoped file rows and drops foreign or stale references", async () => {
|
||||
const { data, deps } = makeFiles()
|
||||
const fileId = await seedFile(data, scope, storageKey)
|
||||
await seedFile(data, otherScope, otherStorageKey, "other-project.txt")
|
||||
const client = stub([
|
||||
{ documentRef: pathname, snippet: "own snippet", score: 0.9, citations: [{ page: 1 }] },
|
||||
{
|
||||
documentRef: `org/${scope.orgId}/${otherStorageKey}`,
|
||||
snippet: "other project snippet",
|
||||
score: 0.8,
|
||||
citations: [],
|
||||
},
|
||||
{
|
||||
documentRef: `org/01900000-0000-7000-0000-000000000777/${storageKey}`,
|
||||
snippet: "other organization snippet",
|
||||
score: 0.7,
|
||||
citations: [],
|
||||
},
|
||||
{
|
||||
documentRef: `org/${scope.orgId}/files/${scope.workspaceId}/${scope.projectId}/01900000-0000-7000-0000-000000000200`,
|
||||
snippet: "stale snippet",
|
||||
score: 0.6,
|
||||
citations: [],
|
||||
},
|
||||
])
|
||||
const service = platformKnowledgeSearch(knowledgeClient(client), deps, { environment: "preview" })
|
||||
|
||||
const hits = await service.search(scope, {
|
||||
collectionName: "contracts",
|
||||
query: "contract",
|
||||
limit: 10,
|
||||
actorId: scope.actor.userId,
|
||||
})
|
||||
|
||||
expect(hits).toEqual([
|
||||
{
|
||||
fileId,
|
||||
filename: "contract.txt",
|
||||
collectionName: "contracts",
|
||||
score: 0.9,
|
||||
chunk: 1,
|
||||
snippet: "own snippet",
|
||||
page: 1,
|
||||
},
|
||||
])
|
||||
expect(client.searches).toHaveLength(1)
|
||||
expect(client.searches[0]).toMatchObject({ collection: "contracts", query: "contract", limit: 10 })
|
||||
expect(client.searches[0]?.tenant).toMatch(/^[0-9a-f]{64}$/)
|
||||
expect(client.searches[0]).not.toHaveProperty("ownerRef")
|
||||
})
|
||||
|
||||
it("refuses a foreign file id before any managed index call", async () => {
|
||||
const { data, deps } = makeFiles()
|
||||
const foreignFileId = await seedFile(data, otherScope, otherStorageKey)
|
||||
const client = stub()
|
||||
const service = platformKnowledgeSearch(knowledgeClient(client), deps, { environment: "preview" })
|
||||
|
||||
await expect(
|
||||
service.index(scope, {
|
||||
collectionName: "contracts",
|
||||
fileId: foreignFileId,
|
||||
filename: "other-project.txt",
|
||||
bytes: new TextEncoder().encode("foreign content"),
|
||||
}),
|
||||
).rejects.toThrow()
|
||||
expect(client.indexed).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe("the managed service's collections", () => {
|
||||
it("creates the collection in the deployment's environment before it indexes: the service answers 404 otherwise", async () => {
|
||||
const { data, deps } = makeFiles()
|
||||
const fileId = await seedFile(data, scope, storageKey)
|
||||
const order: string[] = []
|
||||
const client: Knowledge = {
|
||||
...knowledgeClient(stub()),
|
||||
createCollection: async (input) => {
|
||||
order.push(`collection ${input.collection} in ${input.environment}`)
|
||||
return input
|
||||
},
|
||||
index: async () => {
|
||||
order.push("index")
|
||||
return { jobId: "job-1" }
|
||||
},
|
||||
}
|
||||
const input = { collectionName: "contracts", fileId, filename: "a.txt", bytes: new TextEncoder().encode("text") }
|
||||
|
||||
await platformKnowledgeSearch(client, deps, { environment: "preview" }).index(scope, input)
|
||||
expect(order).toEqual(["collection contracts in preview", "index"])
|
||||
|
||||
order.length = 0
|
||||
const missing = platformKnowledgeSearch(client, deps, { environment: undefined })
|
||||
expect(await missing.readiness(scope)).toMatchObject({ state: "unavailable", refusal: "configuration_missing" })
|
||||
await expect(missing.index(scope, input)).rejects.toMatchObject({
|
||||
reason: "configuration_missing",
|
||||
permanent: "configuration_missing",
|
||||
})
|
||||
expect(order).toEqual([])
|
||||
})
|
||||
|
||||
it("answers no results for a collection nothing was indexed into yet", async () => {
|
||||
const client: Knowledge = {
|
||||
...knowledgeClient(stub()),
|
||||
search: async () => {
|
||||
throw new KnowledgeError("unavailable_resource", "no such collection")
|
||||
},
|
||||
}
|
||||
const hits = await platformKnowledgeSearch(client, makeFiles().deps, { environment: "preview" }).search(scope, {
|
||||
collectionName: "empty",
|
||||
query: "anything",
|
||||
limit: 5,
|
||||
actorId: scope.actor.userId,
|
||||
})
|
||||
expect(hits).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe("the managed service's refusals", () => {
|
||||
const answering = (over: Partial<Knowledge>) => ({ ...knowledgeClient(stub()), ...over })
|
||||
|
||||
it.each([
|
||||
["plan_required", "disabled", "plan_required"],
|
||||
["feature_disabled", "disabled", "feature_disabled"],
|
||||
["forbidden", "unavailable", "scope_missing"],
|
||||
["unauthenticated", "unavailable", "runtime_unavailable"],
|
||||
["runtime_unavailable", "unavailable", "runtime_unavailable"],
|
||||
["indexing", "indexing", "indexing"],
|
||||
] as const)("readiness refused with %s is state %s and the typed refusal %s", async (reason, state, refusal) => {
|
||||
const client = answering({
|
||||
readiness: async () => {
|
||||
throw new KnowledgeError(reason, "the service's own words")
|
||||
},
|
||||
})
|
||||
const answer = await platformKnowledgeSearch(client, makeFiles().deps, { environment: "preview" }).readiness(scope)
|
||||
expect(answer).toEqual({ state, reasons: [], refusal })
|
||||
})
|
||||
|
||||
it("records a refusal waiting cannot fix on the document, and leaves a transient failure to the queue", async () => {
|
||||
const { data, deps } = makeFiles()
|
||||
const fileId = await seedFile(data, scope, storageKey)
|
||||
const bytes = new TextEncoder().encode("managed knowledge")
|
||||
const input = { collectionName: "contracts", fileId, filename: "contract.txt", bytes }
|
||||
|
||||
const quota = platformKnowledgeSearch(
|
||||
answering({
|
||||
job: async (jobId) => ({ jobId, status: "failed", progress: null, generation: null, error: "quota-exhausted" }),
|
||||
}),
|
||||
deps,
|
||||
{ environment: "preview" },
|
||||
)
|
||||
await expect(quota.index(scope, input)).rejects.toMatchObject({ permanent: "quota_exhausted" })
|
||||
|
||||
const scope_missing = platformKnowledgeSearch(
|
||||
answering({
|
||||
index: async () => {
|
||||
throw new KnowledgeError("forbidden", "the token lacks knowledge.collections:use")
|
||||
},
|
||||
}),
|
||||
deps,
|
||||
{ environment: "preview" },
|
||||
)
|
||||
await expect(scope_missing.index(scope, input)).rejects.toMatchObject({ permanent: "scope_missing" })
|
||||
|
||||
const down = platformKnowledgeSearch(
|
||||
answering({
|
||||
index: async () => {
|
||||
throw new KnowledgeError("runtime_unavailable", "try later", true)
|
||||
},
|
||||
}),
|
||||
deps,
|
||||
{ environment: "preview" },
|
||||
)
|
||||
const transient = await down.index(scope, input).catch((error: unknown) => error)
|
||||
expect(transient).toBeInstanceOf(KnowledgeError)
|
||||
expect(transient).not.toHaveProperty("permanent")
|
||||
})
|
||||
})
|
||||
|
||||
describe("bounded durable index polling", () => {
|
||||
it("stops a queued wait on cancellation without cancelling the durable platform job", async () => {
|
||||
const { data, deps } = makeFiles()
|
||||
const fileId = await seedFile(data, scope, storageKey)
|
||||
const controller = new AbortController()
|
||||
let polls = 0
|
||||
let cancellations = 0
|
||||
const client: Knowledge = {
|
||||
...knowledgeClient(stub()),
|
||||
job: async (jobId) => {
|
||||
polls++
|
||||
controller.abort()
|
||||
return { jobId, status: "queued", progress: null, generation: null, error: null }
|
||||
},
|
||||
cancel: async () => {
|
||||
cancellations++
|
||||
},
|
||||
}
|
||||
await expect(
|
||||
platformKnowledgeSearch(client, deps, { environment: "preview", intervalMs: 1 }).index(scope, {
|
||||
collectionName: "contracts",
|
||||
fileId,
|
||||
filename: "a.txt",
|
||||
bytes: new Uint8Array(),
|
||||
signal: controller.signal,
|
||||
}),
|
||||
).rejects.toBeDefined()
|
||||
expect(polls).toBe(1)
|
||||
expect(cancellations).toBe(0)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,144 @@
|
||||
// Every knowledge procedure a page calls, run as the kernel runs it: `createCaller` hands a handler the
|
||||
// container of its kind, and that container refuses a port above the kind. `knowledge.status` backs the
|
||||
// knowledge page and `knowledge.settings.get` the organization's settings page, so a handler that reaches
|
||||
// for an effect port from a read is a 500 `kind_violation` on those pages on every real stack. The ports
|
||||
// stay effects: what changes is what each handler asks for.
|
||||
|
||||
import { Container, type ProcedureKind, type TenantScope } from "@conusai-kit/core"
|
||||
import { createMemoryDataSource, DATA_SOURCE, OBJECT_STORE } from "@conusai-kit/data"
|
||||
import { files } from "@conusai-kit/files"
|
||||
import { fakeObjectStore } from "@conusai-kit/testing"
|
||||
import { describe, expect, it, vi } from "vitest"
|
||||
import {
|
||||
knowledgeStatusImpl,
|
||||
settingsDisableDocumentAiImpl,
|
||||
settingsEnableDocumentAiImpl,
|
||||
settingsGetImpl,
|
||||
} from "../adapters/procedures.js"
|
||||
import { platformKnowledgeReadiness } from "../adapters/readiness.js"
|
||||
import { KNOWLEDGE_ENTITIES } from "../contract/index.js"
|
||||
import {
|
||||
type DocumentAiPlatform,
|
||||
KNOWLEDGE_ENABLEMENT,
|
||||
KNOWLEDGE_READINESS,
|
||||
KNOWLEDGE_SEARCH,
|
||||
type KnowledgeScope,
|
||||
} from "../core/ports.js"
|
||||
|
||||
const scope: TenantScope & KnowledgeScope = {
|
||||
level: "project",
|
||||
actor: { userId: "01900000-0000-7000-0000-000000000009" },
|
||||
orgId: "01900000-0000-7000-0000-000000000001",
|
||||
workspaceId: "01900000-0000-7000-0000-000000000002",
|
||||
projectId: "01900000-0000-7000-0000-000000000003",
|
||||
}
|
||||
const projectInput = { org: "acme", workspace: "general", project: "demo" }
|
||||
const orgInput = { org: "acme" }
|
||||
|
||||
const processContainer = () => {
|
||||
const container = new Container()
|
||||
container.value(DATA_SOURCE, createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES, files] }))
|
||||
container.value(OBJECT_STORE, fakeObjectStore())
|
||||
return container
|
||||
}
|
||||
|
||||
/** A handler as the kernel calls it (`createCaller`): with the container of its procedure's kind. */
|
||||
const run = (
|
||||
impl: { readonly kind: ProcedureKind; readonly handler: (call: never) => unknown },
|
||||
input: unknown,
|
||||
container: Container,
|
||||
) => impl.handler({ input, ctx: { container: container.restrict(impl.kind), scope } } as never)
|
||||
|
||||
const neverBuilt = () =>
|
||||
vi.fn((): never => {
|
||||
throw new Error("a port above this procedure's kind was built")
|
||||
})
|
||||
|
||||
const platform: DocumentAiPlatform = {
|
||||
checkEligibility: async () => ({ ok: true }),
|
||||
indexingQuota: async () => 1000,
|
||||
}
|
||||
|
||||
describe("the ports that leave the process", () => {
|
||||
it("stay refused to a read and a write", () => {
|
||||
const container = processContainer()
|
||||
container.value(KNOWLEDGE_ENABLEMENT, platform)
|
||||
container.register(KNOWLEDGE_SEARCH, neverBuilt())
|
||||
for (const kind of ["read", "write"] as const) {
|
||||
expect(() => container.restrict(kind).resolve(KNOWLEDGE_SEARCH)).toThrow(/cannot resolve knowledge.search/)
|
||||
expect(() => container.restrict(kind).resolve(KNOWLEDGE_ENABLEMENT)).toThrow(
|
||||
/cannot resolve knowledge.enablement/,
|
||||
)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe("the knowledge page's knowledge.status", () => {
|
||||
it("answers through a read handler's container, and never builds the search service", async () => {
|
||||
const container = processContainer()
|
||||
const search = neverBuilt()
|
||||
container.register(KNOWLEDGE_SEARCH, search)
|
||||
await expect(run(knowledgeStatusImpl, projectInput, container)).resolves.toMatchObject({
|
||||
documents: { queued: 0, indexing: 0, indexed: 0, failed: 0, cancelled: 0 },
|
||||
})
|
||||
expect(search).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
|
||||
describe("the settings page's knowledge.settings", () => {
|
||||
it("get answers through a read handler's container, without the enablement platform", async () => {
|
||||
const container = processContainer()
|
||||
const enablement = neverBuilt()
|
||||
container.register(KNOWLEDGE_ENABLEMENT, enablement)
|
||||
await expect(run(settingsGetImpl, orgInput, container)).resolves.toEqual({
|
||||
enabled: false,
|
||||
backend: null,
|
||||
})
|
||||
expect(enablement).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it("enable reaches the platform and stores the configuration", async () => {
|
||||
const container = processContainer()
|
||||
container.value(KNOWLEDGE_ENABLEMENT, platform)
|
||||
await expect(run(settingsEnableDocumentAiImpl, orgInput, container)).resolves.toEqual({
|
||||
enabled: true,
|
||||
backend: "platform",
|
||||
})
|
||||
await expect(run(settingsGetImpl, orgInput, container)).resolves.toEqual({ enabled: true, backend: "platform" })
|
||||
})
|
||||
|
||||
it("disable only flips the stored flag, without the enablement platform", async () => {
|
||||
const container = processContainer()
|
||||
const enablement = neverBuilt()
|
||||
container.register(KNOWLEDGE_ENABLEMENT, enablement)
|
||||
await expect(run(settingsDisableDocumentAiImpl, orgInput, container)).resolves.toEqual({ enabled: false })
|
||||
expect(enablement).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
|
||||
describe("what knowledge.status says of the backend", () => {
|
||||
it("says what the bound readiness says, and asks it with the request's scope", async () => {
|
||||
const container = processContainer()
|
||||
const readiness = vi.fn(async () => ({ state: "degraded" as const, reasons: ["the index is behind"] }))
|
||||
container.value(KNOWLEDGE_READINESS, { readiness })
|
||||
await expect(run(knowledgeStatusImpl, projectInput, container)).resolves.toMatchObject({
|
||||
service: { state: "degraded", reasons: ["the index is behind"] },
|
||||
})
|
||||
expect(readiness).toHaveBeenCalledWith(scope)
|
||||
})
|
||||
|
||||
it("says the backend is not bound in a process that has none", async () => {
|
||||
await expect(run(knowledgeStatusImpl, projectInput, processContainer())).resolves.toMatchObject({
|
||||
service: { state: "unavailable", reasons: ["the knowledge search service is not bound in this process"] },
|
||||
})
|
||||
})
|
||||
|
||||
it("reports the managed service failing to answer as unavailable, not as an error page", async () => {
|
||||
const container = processContainer()
|
||||
const client = { readiness: vi.fn(async () => Promise.reject(new Error("no answer\nstack"))) }
|
||||
container.value(KNOWLEDGE_READINESS, platformKnowledgeReadiness(client as never))
|
||||
await expect(run(knowledgeStatusImpl, projectInput, container)).resolves.toMatchObject({
|
||||
service: { state: "unavailable", reasons: [], refusal: "runtime_unavailable" },
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,157 @@
|
||||
import { syntheticNativeReadiness } from "../../../test/support/native-readiness.js"
|
||||
// A refusal of the platform's knowledge service reaches a person as one typed code: the procedures end to
|
||||
// end, on the real module, with the service's client refusing underneath; and each code a form looks up.
|
||||
|
||||
import { AppError, buildRegistry, Container, createCaller, type TransportCaller } from "@conusai-kit/core"
|
||||
import { createMemoryDataSource, DATA_SOURCE, entitiesOf, OBJECT_STORE, SYSTEM_DATA } from "@conusai-kit/data"
|
||||
import { FILES_REGISTRY, filesModule } from "@conusai-kit/files"
|
||||
import { createI18n } from "@conusai-kit/i18n"
|
||||
import { KnowledgeError, type KnowledgeReasonCode } from "@conusai-kit/platform"
|
||||
import { createTenancyHooks, TENANCY, tenancyModule, users } from "@conusai-kit/tenancy"
|
||||
import { fakeObjectStore } from "@conusai-kit/testing"
|
||||
import { describe, expect, it } from "vitest"
|
||||
import { knowledgeRefusal } from "../adapters/refusals.js"
|
||||
import { KNOWLEDGE_REFUSALS, type KnowledgeRefusal } from "../contract/index.js"
|
||||
import {
|
||||
KNOWLEDGE_ENABLEMENT,
|
||||
KNOWLEDGE_FLAGS,
|
||||
KNOWLEDGE_READINESS,
|
||||
KNOWLEDGE_SEARCH,
|
||||
type KnowledgeSearchService,
|
||||
} from "../core/ports.js"
|
||||
import { knowledgeModule } from "../index.js"
|
||||
import { refusalLabels, refusalOf } from "../ui/labels.js"
|
||||
|
||||
async function app(
|
||||
service: KnowledgeSearchService,
|
||||
options: { flags?: boolean } = {},
|
||||
): Promise<{ call: TransportCaller; slugs: Record<string, string> }> {
|
||||
const registry = buildRegistry([tenancyModule, filesModule, knowledgeModule])
|
||||
const data = createMemoryDataSource({ entities: entitiesOf(registry, "postgres") })
|
||||
const root = new Container()
|
||||
root.value(DATA_SOURCE, data)
|
||||
root.value(SYSTEM_DATA, data)
|
||||
root.value(OBJECT_STORE, fakeObjectStore())
|
||||
root.value(FILES_REGISTRY, registry)
|
||||
root.value(KNOWLEDGE_SEARCH, service)
|
||||
root.value(KNOWLEDGE_READINESS, service)
|
||||
root.value(KNOWLEDGE_ENABLEMENT, { checkEligibility: async () => ({ ok: true }), indexingQuota: async () => 1000 })
|
||||
if (options.flags !== false) root.value(KNOWLEDGE_FLAGS, { enabled: () => true })
|
||||
const user = await data.global(users).insert({
|
||||
issuer: "https://id.example.test",
|
||||
subject: "owner",
|
||||
email: "owner@example.test",
|
||||
displayName: "Owner",
|
||||
locale: "en",
|
||||
})
|
||||
const container = root.child()
|
||||
const tenancy = createTenancyHooks({ data, registry })
|
||||
container.value(TENANCY, tenancy)
|
||||
const call = createCaller(
|
||||
{ registry, container, tenancy, serviceReadiness: syntheticNativeReadiness },
|
||||
{ actor: { userId: user.id } },
|
||||
)
|
||||
await call.callByName("tenancy.org.create", { slug: "acme", name: "Acme" })
|
||||
await call.callByName("tenancy.workspace.create", { org: "acme", slug: "ws", name: "WS", visibility: "org" })
|
||||
await call.callByName("tenancy.project.create", {
|
||||
org: "acme",
|
||||
workspace: "ws",
|
||||
slug: "p",
|
||||
name: "P",
|
||||
moduleKey: "knowledge",
|
||||
})
|
||||
return { call, slugs: { org: "acme", workspace: "ws", project: "p" } }
|
||||
}
|
||||
|
||||
const refusing = (reason: KnowledgeReasonCode): KnowledgeSearchService => ({
|
||||
readiness: async () => ({ state: "disabled", reasons: [], refusal: "plan_required" }),
|
||||
index: async () => undefined,
|
||||
search: async () => {
|
||||
throw new KnowledgeError(reason, "the service's own words")
|
||||
},
|
||||
remove: async () => undefined,
|
||||
})
|
||||
|
||||
describe("a refused search", () => {
|
||||
it.each([
|
||||
["plan_required", "plan_required", 403],
|
||||
["feature_disabled", "feature_disabled", 403],
|
||||
["quota_exhausted", "quota_exhausted", 429],
|
||||
["budget_exhausted", "budget_exhausted", 429],
|
||||
["forbidden", "scope_missing", 503],
|
||||
["indexing", "indexing", 503],
|
||||
["approval_required", "approval_required", 403],
|
||||
["runtime_unavailable", "runtime_unavailable", 503],
|
||||
["unauthenticated", "runtime_unavailable", 503],
|
||||
] as const)("by %s answers %s", async (reason, code, status) => {
|
||||
const { call, slugs } = await app(refusing(reason))
|
||||
await call.callByName("knowledge.collection.create", { ...slugs, name: "contracts" })
|
||||
const refused = await call
|
||||
.callByName("knowledge.search", { ...slugs, collectionName: "contracts", query: "anything" })
|
||||
.catch((error: unknown) => error)
|
||||
expect(refused).toBeInstanceOf(AppError)
|
||||
expect((refused as AppError).problem).toMatchObject({ code, status })
|
||||
})
|
||||
|
||||
it("leaves a resource the service does not hold as not found, and a programming error as it is", () => {
|
||||
const missing = knowledgeRefusal(new KnowledgeError("unavailable_resource", "no such collection"))
|
||||
expect((missing as AppError).problem.status).toBe(404)
|
||||
const bug = new TypeError("not a refusal")
|
||||
expect(knowledgeRefusal(bug)).toBe(bug)
|
||||
})
|
||||
})
|
||||
|
||||
describe("the status a page reads", () => {
|
||||
it("carries the service's refusal through the read procedure", async () => {
|
||||
const { call, slugs } = await app(refusing("plan_required"))
|
||||
expect(await call.callByName("knowledge.status", slugs)).toMatchObject({
|
||||
enabled: true,
|
||||
service: { state: "disabled", reasons: [], refusal: "plan_required" },
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe("enabling document AI", () => {
|
||||
const ready: KnowledgeSearchService = {
|
||||
readiness: async () => ({ state: "ready", reasons: [] }),
|
||||
index: async () => undefined,
|
||||
search: async () => [],
|
||||
remove: async () => undefined,
|
||||
}
|
||||
|
||||
it("turns knowledge on for the organization's projects, and disabling turns it off again", async () => {
|
||||
const { call, slugs } = await app(ready, { flags: false })
|
||||
const enabled = async () => ((await call.callByName("knowledge.status", slugs)) as { enabled: boolean }).enabled
|
||||
expect(await enabled()).toBe(false)
|
||||
await call.callByName("knowledge.settings.enableDocumentAi", { org: slugs.org })
|
||||
expect(await enabled()).toBe(true)
|
||||
await call.callByName("knowledge.settings.disableDocumentAi", { org: slugs.org })
|
||||
expect(await enabled()).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe("a form's refusal", () => {
|
||||
const t = (key: string) => key
|
||||
it("names every typed refusal, and leaves any other error to the form's own text", () => {
|
||||
const labels = refusalLabels(t)
|
||||
expect(Object.keys(labels).sort()).toEqual([...KNOWLEDGE_REFUSALS].sort())
|
||||
const refused = (code: KnowledgeRefusal) => ({ problem: { code } })
|
||||
expect(refusalOf(labels, refused("scope_missing"))).toEqual({
|
||||
code: "scope_missing",
|
||||
text: "refusals.scope_missing",
|
||||
})
|
||||
expect(refusalOf(labels, { problem: { code: "not_found" } })).toBeUndefined()
|
||||
expect(refusalOf(labels, new Error("boom"))).toBeUndefined()
|
||||
})
|
||||
|
||||
it("has a message in both languages, the Lithuanian one its own", () => {
|
||||
const catalogues = buildRegistry([tenancyModule, filesModule, knowledgeModule]).catalogues
|
||||
const en = createI18n("en", { en: { knowledge: catalogues.en?.knowledge ?? {} } })
|
||||
const lt = createI18n("lt", { lt: { knowledge: catalogues.lt?.knowledge ?? {} } })
|
||||
for (const code of KNOWLEDGE_REFUSALS) {
|
||||
const english = en.message(`knowledge.refusals.${code}`)
|
||||
expect(english, code).not.toContain("refusals.")
|
||||
expect(lt.message(`knowledge.refusals.${code}`), code).not.toBe(english)
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,121 @@
|
||||
// The enablement flow's semantics: idempotent enable, typed denials from the platform checks, nothing
|
||||
// provisioned on the app's side, and disable that preserves every row and every indexed document.
|
||||
|
||||
import type { TenantScope } from "@conusai-kit/core"
|
||||
import { createMemoryDataSource } from "@conusai-kit/data"
|
||||
import { DEFAULT_FILES_POLICY } from "@conusai-kit/files"
|
||||
import { fakeObjectStore } from "@conusai-kit/testing"
|
||||
import { describe, expect, it } from "vitest"
|
||||
import { collections, config, documents, KNOWLEDGE_ENTITIES } from "../contract/index.js"
|
||||
import { createCollection } from "../core/collections.js"
|
||||
import type { DocumentAiPlatform } from "../core/ports.js"
|
||||
import { disableDocumentAi, enableDocumentAi, getSettings } from "../core/settings.js"
|
||||
|
||||
const ORG = "01900000-0000-7000-0000-000000000001"
|
||||
const scope: TenantScope = { level: "org", actor: { userId: "01900000-0000-7000-0000-000000000009" }, orgId: ORG }
|
||||
|
||||
const platform = (over: Partial<DocumentAiPlatform> = {}): DocumentAiPlatform => ({
|
||||
checkEligibility: async () => ({ ok: true }),
|
||||
indexingQuota: async () => 1000,
|
||||
...over,
|
||||
})
|
||||
|
||||
const deps = (data: ReturnType<typeof createMemoryDataSource>, p: DocumentAiPlatform) => ({
|
||||
data,
|
||||
platform: p,
|
||||
now: () => Date.UTC(2026, 9, 23),
|
||||
})
|
||||
|
||||
describe("the enablement flow", () => {
|
||||
it("enables idempotently: the second call writes nothing and answers the same configuration", async () => {
|
||||
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES] })
|
||||
const first = await enableDocumentAi(deps(data, platform()), scope)
|
||||
expect(first).toEqual({ enabled: true, backend: "platform" })
|
||||
const second = await enableDocumentAi(deps(data, platform()), scope)
|
||||
expect(second).toEqual(first)
|
||||
expect((await getSettings(deps(data, platform()), scope)).enabled).toBe(true)
|
||||
})
|
||||
|
||||
it("surfaces the platform's denials as typed reason codes, retirable", async () => {
|
||||
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES] })
|
||||
const denied = await enableDocumentAi(
|
||||
deps(data, platform({ checkEligibility: async () => ({ ok: false, reason: "plan_required" }) })),
|
||||
scope,
|
||||
).catch((error: unknown) => error)
|
||||
expect((denied as { code?: string }).code).toBe("plan_required")
|
||||
// Nothing was persisted by the denied attempt; the retry path runs the same checks again.
|
||||
expect((await getSettings(deps(data, platform()), scope)).enabled).toBe(false)
|
||||
const quota = await enableDocumentAi(deps(data, platform({ indexingQuota: async () => 0 })), scope).catch(
|
||||
(error: unknown) => error,
|
||||
)
|
||||
expect((quota as { code?: string }).code).toBe("quota_exhausted")
|
||||
})
|
||||
|
||||
it("provisions nothing of the app's own: enabling records that the platform serves this organization", async () => {
|
||||
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES] })
|
||||
const asked: string[] = []
|
||||
const watched: DocumentAiPlatform = new Proxy(platform(), {
|
||||
get: (target, name, receiver) => {
|
||||
asked.push(String(name))
|
||||
return Reflect.get(target, name, receiver)
|
||||
},
|
||||
})
|
||||
await enableDocumentAi(deps(data, watched), scope)
|
||||
expect(asked).toEqual(["checkEligibility", "indexingQuota"])
|
||||
const rows = await data.forScope(scope).of(config).list({ limit: 1 })
|
||||
expect(rows.items[0]).toMatchObject({ enabled: true, backend: "platform", serviceId: null, url: null })
|
||||
expect(await getSettings(deps(data, platform()), scope)).toEqual({ enabled: true, backend: "platform" })
|
||||
})
|
||||
|
||||
it("disable preserves collections, documents and the recorded configuration; re-enable resumes without duplication", async () => {
|
||||
const data = createMemoryDataSource({ entities: [...KNOWLEDGE_ENTITIES] })
|
||||
const projectScope: TenantScope = {
|
||||
level: "project",
|
||||
actor: { userId: "01900000-0000-7000-0000-000000000009" },
|
||||
orgId: ORG,
|
||||
workspaceId: "01900000-0000-7000-0000-000000000002",
|
||||
projectId: "01900000-0000-7000-0000-000000000003",
|
||||
}
|
||||
const knowledgeDepsForProject = {
|
||||
data,
|
||||
files: { data, store: fakeObjectStore(), policy: DEFAULT_FILES_POLICY },
|
||||
flags: { enabled: async () => true },
|
||||
}
|
||||
await enableDocumentAi(deps(data, platform()), scope)
|
||||
const collection = await createCollection(knowledgeDepsForProject, projectScope as never, "contracts")
|
||||
expect(collection.name).toBe("contracts")
|
||||
// Seed a document row directly (the upload half is the files module's own suite).
|
||||
await data.forScope(projectScope).of(documents).insert({
|
||||
collectionName: "contracts",
|
||||
fileId: "01900000-0000-7000-0000-000000000099",
|
||||
filename: "a.txt",
|
||||
status: "indexed",
|
||||
error: null,
|
||||
indexedAt: "2026-10-23T00:00:00.000Z",
|
||||
requestedBy: "01900000-0000-7000-0000-000000000009",
|
||||
})
|
||||
|
||||
const disabled = await disableDocumentAi(deps(data, platform()), scope)
|
||||
expect(disabled.enabled).toBe(false)
|
||||
// Everything is still there.
|
||||
expect((await getSettings(deps(data, platform()), scope)).backend).toBe("platform")
|
||||
const collectionsAfter = await data
|
||||
.forScope(projectScope)
|
||||
.of(collections)
|
||||
.list({ where: [{ field: "name", op: "eq", value: "contracts" }], limit: 1 })
|
||||
expect(collectionsAfter.items).toHaveLength(1)
|
||||
const documentsAfter = await data
|
||||
.forScope(projectScope)
|
||||
.of(documents)
|
||||
.list({ where: [{ field: "collectionName", op: "eq", value: "contracts" }], limit: 1 })
|
||||
expect(documentsAfter.items[0]?.status).toBe("indexed")
|
||||
|
||||
// Re-enable: idempotent, same row, no second collection.
|
||||
await enableDocumentAi(deps(data, platform()), scope)
|
||||
const again = await data
|
||||
.forScope(projectScope)
|
||||
.of(collections)
|
||||
.list({ where: [{ field: "name", op: "eq", value: "contracts" }], limit: 1 })
|
||||
expect(again.items).toHaveLength(1)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,86 @@
|
||||
// The components' markup, rendered on the server the way a page renders it: what a screen reader
|
||||
// and a keyboard user get is in the HTML — a name for every control, a status that says what a
|
||||
// colour would, and the AI disclosure where AI output appears.
|
||||
|
||||
import { buildRegistry } from "@conusai-kit/core"
|
||||
import { builtInCatalogues, createI18n, uiLabels } from "@conusai-kit/i18n"
|
||||
import { createDataClient, DataProvider } from "@conusai-kit/ui/data"
|
||||
import { UiLabelsProvider } from "@conusai-kit/ui/primitives"
|
||||
import { renderToStaticMarkup } from "react-dom/server"
|
||||
import { describe, expect, it, vi } from "vitest"
|
||||
import { appModules } from "../../index.js"
|
||||
import { searchFormLabels } from "../ui/labels.js"
|
||||
import { SearchForm } from "../ui/search-form.js"
|
||||
import { SettingsSection } from "../ui/settings-section.js"
|
||||
|
||||
const catalogues = buildRegistry([...appModules]).catalogues
|
||||
const i18n = createI18n("en", { en: { knowledge: catalogues.en?.knowledge ?? {} } })
|
||||
const t = (key: string, values?: Record<string, string | number>) => i18n.message(`knowledge.${key}`, values)
|
||||
const failed = "That did not work: {detail}"
|
||||
const slugs = { org: "acme", workspace: "ws", project: "p" }
|
||||
|
||||
const kitStrings = uiLabels(createI18n("en", builtInCatalogues))
|
||||
|
||||
const inProvider = (node: React.ReactNode) =>
|
||||
renderToStaticMarkup(
|
||||
<UiLabelsProvider labels={kitStrings}>
|
||||
<DataProvider client={createDataClient({ fetch: async () => new Response(null, { status: 501 }) })}>
|
||||
{node}
|
||||
</DataProvider>
|
||||
</UiLabelsProvider>,
|
||||
)
|
||||
|
||||
describe("the search form", () => {
|
||||
const html = inProvider(
|
||||
<SearchForm slugs={slugs} collectionName="contracts" labels={searchFormLabels(t)} locale="en" />,
|
||||
)
|
||||
|
||||
it("names its controls and carries the AI disclosure", () => {
|
||||
expect(html).toMatch(/<label[^>]*for=/)
|
||||
expect(html).toContain('type="search"')
|
||||
expect(html).toContain("kit-ai-disclosure")
|
||||
expect(html).toContain(t("search.disclosure"))
|
||||
})
|
||||
})
|
||||
|
||||
describe("the settings section", () => {
|
||||
const labels = {
|
||||
heading: t("settings.heading"),
|
||||
state: t("settings.state"),
|
||||
stateOn: t("settings.stateOn"),
|
||||
stateOff: t("settings.stateOff"),
|
||||
backend: t("settings.backend"),
|
||||
backendPlatform: t("settings.backendPlatform"),
|
||||
enable: t("settings.enable"),
|
||||
enabling: t("settings.enabling"),
|
||||
disable: t("settings.disable"),
|
||||
disabling: t("settings.disabling"),
|
||||
enabledDone: t("settings.enabledDone"),
|
||||
disabledDone: t("settings.disabledDone"),
|
||||
planRequired: t("settings.planRequired"),
|
||||
featureDisabled: t("settings.featureDisabled"),
|
||||
quotaExhausted: t("settings.quotaExhausted"),
|
||||
upgrade: t("settings.upgrade"),
|
||||
hint: t("settings.hint"),
|
||||
failed,
|
||||
}
|
||||
|
||||
it("renders the state and the enable action when disabled", () => {
|
||||
const html = inProvider(<SettingsSection org="acme" initial={{ enabled: false, backend: null }} labels={labels} />)
|
||||
expect(html).toContain(t("settings.stateOff"))
|
||||
expect(html).toContain(`>${t("settings.enable")}<`)
|
||||
})
|
||||
|
||||
it("says the platform serves the organization, with the disable action, when enabled", () => {
|
||||
const html = inProvider(
|
||||
<SettingsSection org="acme" initial={{ enabled: true, backend: "platform" }} labels={labels} />,
|
||||
)
|
||||
expect(html).toContain(t("settings.backendPlatform").replaceAll("'", "'"))
|
||||
expect(html).toContain(`>${t("settings.disable")}<`)
|
||||
})
|
||||
})
|
||||
|
||||
// Explicit synthetic admission: module-local fixtures exercise the selected module.
|
||||
vi.mock("../../inclusion.json", () => ({
|
||||
default: { schemaVersion: 1, selected: ["knowledge"], everAdopted: ["knowledge"] },
|
||||
}))
|
||||
@@ -0,0 +1,61 @@
|
||||
// The automatic intake's enabling-person metadata is an account-data share. The
|
||||
// account kernel supplies the subject; this has no public procedure or bearer.
|
||||
import type { TenantScope } from "@conusai-kit/core"
|
||||
import { systemActor } from "@conusai-kit/core/worker"
|
||||
import { DATA_SOURCE, SYSTEM_DATA } from "@conusai-kit/data"
|
||||
import { type AccountDataContext, type AccountDataContribution, servedProjects } from "@conusai-kit/tenancy"
|
||||
import { autoIndex } from "../contract/index.js"
|
||||
|
||||
async function* ownedOptIns(context: AccountDataContext) {
|
||||
context.signal?.throwIfAborted()
|
||||
const data = context.container.resolve(DATA_SOURCE)
|
||||
const system = context.container.resolve(SYSTEM_DATA)
|
||||
for await (const project of servedProjects(system)) {
|
||||
context.signal?.throwIfAborted()
|
||||
const scope: TenantScope = { level: "project", ...project, actor: systemActor("knowledge.account") }
|
||||
for (let cursor: string | undefined; ; ) {
|
||||
const page = await data
|
||||
.forScope(scope)
|
||||
.of(autoIndex)
|
||||
.list({
|
||||
where: [{ field: "enabledBy", op: "eq", value: context.subject }],
|
||||
limit: 100,
|
||||
...(cursor === undefined ? {} : { cursor }),
|
||||
})
|
||||
for (const row of page.items) yield { scope, row }
|
||||
if (!page.hasMore || page.cursor === undefined) break
|
||||
cursor = page.cursor
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export const knowledgeAccountData: AccountDataContribution = {
|
||||
key: "knowledge",
|
||||
async export(context) {
|
||||
const automaticIndexing = []
|
||||
for await (const { scope, row } of ownedOptIns(context)) {
|
||||
automaticIndexing.push({
|
||||
orgId: scope.orgId,
|
||||
workspaceId: scope.workspaceId,
|
||||
projectId: scope.projectId,
|
||||
collectionName: row.collectionName,
|
||||
enabledAt: row.enabledAt,
|
||||
})
|
||||
}
|
||||
return { automaticIndexing }
|
||||
},
|
||||
async erase(context) {
|
||||
const data = context.container.resolve(DATA_SOURCE)
|
||||
for await (const { scope, row } of ownedOptIns(context)) {
|
||||
context.signal?.throwIfAborted()
|
||||
await data.unitOfWork.run(scope, async (tx) => {
|
||||
const repository = tx.of(autoIndex)
|
||||
const current = repository.lock ? await repository.lock(row.id) : await repository.get(row.id)
|
||||
// A different administrator may have replaced the opt-in since discovery.
|
||||
if (!current || current.enabledBy !== context.subject) return
|
||||
if (!repository.lock) await repository.update(current.id, {}, current.version)
|
||||
await repository.delete(current.id)
|
||||
})
|
||||
}
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
// The enablement platform over the platform's own APIs, server-side. Eligibility reads the
|
||||
// rollout flags (agent_document_tools, agent_knowledge_tools) from the flag snapshot and probes
|
||||
// the AI tools registry for the document tool family — discovery is the cheap plan/binding/flag
|
||||
// filter, and "a working OCR endpoint does not by itself make OCR a discoverable tool". Nothing is
|
||||
// provisioned for the app: the platform's knowledge service owns the index (ADR-061). The quota
|
||||
// pre-check has no platform API yet (the metering slice owns it); the indexing calls themselves
|
||||
// carry the typed quota_exhausted refusal at run time.
|
||||
|
||||
import { createRegistryClient, ToolIndexingError } from "@conusai-sdk/ai"
|
||||
import type { EnablementReason } from "../contract/index.js"
|
||||
import type { DocumentAiPlatform } from "../core/ports.js"
|
||||
|
||||
export const AGENT_DOCUMENT_TOOLS_FLAG = "agent_document_tools"
|
||||
export const AGENT_KNOWLEDGE_TOOLS_FLAG = "agent_knowledge_tools"
|
||||
|
||||
const FLAG_KEYS = [AGENT_DOCUMENT_TOOLS_FLAG, AGENT_KNOWLEDGE_TOOLS_FLAG] as const
|
||||
|
||||
export interface SdkDocumentAiPlatformOptions {
|
||||
/** The deployment contract's API URL and token. */
|
||||
apiUrl: string
|
||||
token: string
|
||||
fetch?: typeof fetch
|
||||
}
|
||||
|
||||
export function sdkDocumentAiPlatform(options: SdkDocumentAiPlatformOptions): DocumentAiPlatform {
|
||||
const http = options.fetch ?? fetch
|
||||
|
||||
const rolloutFlagsOff = async (): Promise<string | null> => {
|
||||
try {
|
||||
const response = await http(new URL(`${options.apiUrl.replace(/\/+$/, "")}/flags/snapshot`), {
|
||||
headers: { authorization: `Bearer ${options.token}`, accept: "application/json" },
|
||||
signal: AbortSignal.timeout(5_000),
|
||||
})
|
||||
if (!response.ok) return null // the flags service is not the eligibility authority
|
||||
const body = (await response.json()) as { flags?: Array<{ key?: string; enabled?: boolean }> }
|
||||
for (const flag of body.flags ?? []) {
|
||||
if (
|
||||
typeof flag.key === "string" &&
|
||||
(FLAG_KEYS as readonly string[]).includes(flag.key) &&
|
||||
flag.enabled === false
|
||||
) {
|
||||
return flag.key
|
||||
}
|
||||
}
|
||||
return null
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
async checkEligibility(): Promise<{ ok: true } | { ok: false; reason: EnablementReason; detail?: string }> {
|
||||
// The rollout flags first: a kill switch or a disabled feature refuses before any probe.
|
||||
const off = await rolloutFlagsOff()
|
||||
if (off !== null) return { ok: false, reason: "feature_disabled", detail: `the ${off} flag is off` }
|
||||
// The plan probe: can this deployment discover the document tool family at all?
|
||||
try {
|
||||
const registry = createRegistryClient({
|
||||
env: { CONUSAI_API_URL: options.apiUrl, CONUSAI_API_TOKEN: options.token },
|
||||
fetch: http,
|
||||
})
|
||||
const found = await registry.tools.search("read and index documents")
|
||||
if (found.length === 0) {
|
||||
return { ok: false, reason: "plan_required", detail: "the ai_tools plan capability lists no document tools" }
|
||||
}
|
||||
return { ok: true }
|
||||
} catch (error) {
|
||||
if (error instanceof ToolIndexingError) {
|
||||
return { ok: false, reason: "feature_disabled", detail: "the tool registry is still indexing" }
|
||||
}
|
||||
return {
|
||||
ok: false,
|
||||
reason: "plan_required",
|
||||
detail: "the AI tools registry did not answer the eligibility probe",
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
async indexingQuota() {
|
||||
// No pre-check API exists on the platform yet; the call-time quota refusal is the hard
|
||||
// gate. Until the limit endpoint lands, the admission check answers "within quota".
|
||||
return Number.MAX_SAFE_INTEGER
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
// The knowledge jobs and the subscribers that enqueue them.
|
||||
//
|
||||
// A web request writes the document row and publishes the outbox event in one transaction; the
|
||||
// module's own subscriber (running in the worker's outbox relay) turns the event into the job,
|
||||
// with the (collection, file) pair as the singleton key — a redelivered event adds no second job.
|
||||
//
|
||||
// The relay delivers every record in its organization's scope, with a system actor, and the worker binds
|
||||
// `KNOWLEDGE_SCOPE` to that delivery scope. The organization comes from there, never from the payload
|
||||
// (decision AT-F6); only the workspace and the project come from the record, which this module or the files
|
||||
// module wrote inside that organization's transaction, as `projectScopeOf` of @conusai-kit/files does. A job,
|
||||
// unlike a delivery, runs in the project scope it was enqueued in.
|
||||
|
||||
import { type EventContext, internal, type JobDef, serviceRefusal, subscribe } from "@conusai-kit/core"
|
||||
import { DATA_SOURCE, JOB_QUEUE } from "@conusai-kit/data"
|
||||
import { fileDeleted, type ProcessorJobPayload, processorJobPayloadSchema } from "@conusai-kit/files"
|
||||
import { NATIVE_JOB_ADMISSION } from "@conusai-kit/platform"
|
||||
import { projects, workspaces } from "@conusai-kit/tenancy"
|
||||
import { z } from "zod"
|
||||
import {
|
||||
AUTO_INDEX_JOB,
|
||||
documentIndexRequested,
|
||||
documentRemoveRequested,
|
||||
INDEX_JOB,
|
||||
REMOVE_JOB,
|
||||
} from "../contract/index.js"
|
||||
import { autoIndexUpload } from "../core/auto-index-job.js"
|
||||
import { requestSourceRemoval } from "../core/documents.js"
|
||||
import { indexDocument, removeDocument } from "../core/index-job.js"
|
||||
import { KNOWLEDGE_SCOPE, type KnowledgeScope } from "../core/ports.js"
|
||||
import { knowledgeDeps, knowledgeSearchDeps } from "./procedures.js"
|
||||
|
||||
export const documentPayload = z.object({
|
||||
documentId: z.string(),
|
||||
collectionName: z.string(),
|
||||
fileId: z.string(),
|
||||
/** A remove job's: the object to remove, resolved when the removal was asked for; absent only on an older record. */
|
||||
storageKey: z.string().optional(),
|
||||
})
|
||||
export type DocumentJobPayload = z.output<typeof documentPayload>
|
||||
|
||||
export const indexJobKey = (collectionName: string, fileId: string): string =>
|
||||
`knowledge.index.${collectionName}.${fileId}`
|
||||
export const removeJobKey = (collectionName: string, fileId: string): string =>
|
||||
`knowledge.remove.${collectionName}.${fileId}`
|
||||
|
||||
/**
|
||||
* The project a delivery acts in: the delivery's organization and actor, and the record's workspace and project.
|
||||
* A record that names another organization is refused here rather than acted on there, and so is one whose workspace
|
||||
* the organization does not have or whose project is not in that workspace: both are read in the delivery's own
|
||||
* organization before anything is enqueued or changed. That refusal can never succeed on a retry; the relay has no
|
||||
* terminal-failure path yet, so it is retried until marked stuck, with the reason (a kit-jobs gap, recorded in the
|
||||
* template lessons). An archived project is acted on: an archive hides a project and does not end its work
|
||||
* (kit-tenancy `suspension.ts`).
|
||||
*/
|
||||
async function projectScopeOf(
|
||||
ctx: EventContext,
|
||||
payload: { orgId: string; workspaceId: string; projectId: string },
|
||||
): Promise<KnowledgeScope & { level: "project" }> {
|
||||
const ambient = ctx.container.resolve(KNOWLEDGE_SCOPE)
|
||||
if (ambient.orgId !== payload.orgId) {
|
||||
throw internal(`a knowledge event of organization ${payload.orgId} was delivered in ${ambient.orgId}`)
|
||||
}
|
||||
const data = ctx.container.resolve(DATA_SOURCE)
|
||||
const inOrg = { level: "org" as const, actor: ambient.actor, orgId: ambient.orgId }
|
||||
if (!(await data.forScope(inOrg).of(workspaces).get(payload.workspaceId))) {
|
||||
throw internal(
|
||||
`a knowledge event names workspace ${payload.workspaceId}, which organization ${ambient.orgId} does not have`,
|
||||
)
|
||||
}
|
||||
const inWorkspace = {
|
||||
level: "workspace" as const,
|
||||
actor: ambient.actor,
|
||||
orgId: ambient.orgId,
|
||||
workspaceId: payload.workspaceId,
|
||||
}
|
||||
if (!(await data.forScope(inWorkspace).of(projects).get(payload.projectId))) {
|
||||
throw internal(
|
||||
`a knowledge event names project ${payload.projectId}, which is not in workspace ${payload.workspaceId}`,
|
||||
)
|
||||
}
|
||||
return {
|
||||
level: "project",
|
||||
actor: ambient.actor,
|
||||
orgId: ambient.orgId,
|
||||
workspaceId: payload.workspaceId,
|
||||
projectId: payload.projectId,
|
||||
}
|
||||
}
|
||||
|
||||
export const indexJob: JobDef<DocumentJobPayload> = {
|
||||
payload: documentPayload,
|
||||
async handler(payload, ctx) {
|
||||
const scope = ctx.container.resolve(KNOWLEDGE_SCOPE)
|
||||
if (scope.level !== "project") throw internal(`a knowledge job runs in a project scope, not a ${scope.level} one`)
|
||||
await indexDocument(
|
||||
{
|
||||
...knowledgeSearchDeps(ctx.container),
|
||||
beforeIndex: async (subject) => {
|
||||
if (!ctx.signal || !ctx.container.has(NATIVE_JOB_ADMISSION)) throw serviceRefusal("observation_missing")
|
||||
ctx.signal.throwIfAborted()
|
||||
await ctx.container.resolve(NATIVE_JOB_ADMISSION).authorize(scope, INDEX_JOB, subject, ctx.signal)
|
||||
},
|
||||
},
|
||||
scope,
|
||||
payload.documentId,
|
||||
{
|
||||
...(ctx.signal === undefined ? {} : { signal: ctx.signal }),
|
||||
...(ctx.finalAttempt === undefined ? {} : { finalAttempt: ctx.finalAttempt }),
|
||||
},
|
||||
)
|
||||
},
|
||||
}
|
||||
|
||||
/**
|
||||
* The automatic PDF intake (core-11): the files module enqueues it for every stored upload this module's
|
||||
* processor wants, in the upload's project. It acts for the administrator who opted the project in, so
|
||||
* the worker's native admission checks that administrator's permission and the platform's services again
|
||||
* before each step that leaves the process, as the index job does for the person who asked.
|
||||
*/
|
||||
export const autoIndexJob: JobDef<ProcessorJobPayload> = {
|
||||
payload: processorJobPayloadSchema,
|
||||
async handler(payload, ctx) {
|
||||
const scope = ctx.container.resolve(KNOWLEDGE_SCOPE)
|
||||
if (scope.level !== "project") throw internal(`a knowledge job runs in a project scope, not a ${scope.level} one`)
|
||||
await autoIndexUpload(
|
||||
{
|
||||
...knowledgeSearchDeps(ctx.container),
|
||||
beforeIndex: async (subject) => {
|
||||
if (!ctx.signal || !ctx.container.has(NATIVE_JOB_ADMISSION)) throw serviceRefusal("observation_missing")
|
||||
ctx.signal.throwIfAborted()
|
||||
await ctx.container.resolve(NATIVE_JOB_ADMISSION).authorize(scope, AUTO_INDEX_JOB, subject, ctx.signal)
|
||||
},
|
||||
},
|
||||
scope,
|
||||
payload,
|
||||
{
|
||||
...(ctx.signal === undefined ? {} : { signal: ctx.signal }),
|
||||
...(ctx.finalAttempt === undefined ? {} : { finalAttempt: ctx.finalAttempt }),
|
||||
},
|
||||
)
|
||||
},
|
||||
}
|
||||
|
||||
export const removeJob: JobDef<DocumentJobPayload> = {
|
||||
payload: documentPayload,
|
||||
async handler(payload, ctx) {
|
||||
const scope = ctx.container.resolve(KNOWLEDGE_SCOPE)
|
||||
if (scope.level !== "project") throw internal(`a knowledge job runs in a project scope, not a ${scope.level} one`)
|
||||
await removeDocument(knowledgeSearchDeps(ctx.container), scope, payload.documentId, payload.storageKey)
|
||||
},
|
||||
}
|
||||
|
||||
async function onDocumentIndexRequested(
|
||||
event: {
|
||||
name: string
|
||||
payload: {
|
||||
orgId: string
|
||||
workspaceId: string
|
||||
projectId: string
|
||||
documentId: string
|
||||
collectionName: string
|
||||
fileId: string
|
||||
}
|
||||
},
|
||||
ctx: EventContext,
|
||||
): Promise<void> {
|
||||
const scope = await projectScopeOf(ctx, event.payload)
|
||||
await ctx.container.resolve(JOB_QUEUE).enqueue(
|
||||
scope,
|
||||
INDEX_JOB,
|
||||
{
|
||||
documentId: event.payload.documentId,
|
||||
collectionName: event.payload.collectionName,
|
||||
fileId: event.payload.fileId,
|
||||
},
|
||||
{
|
||||
key: indexJobKey(event.payload.collectionName, event.payload.fileId),
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
async function onDocumentRemoveRequested(
|
||||
event: {
|
||||
name: string
|
||||
payload: {
|
||||
orgId: string
|
||||
workspaceId: string
|
||||
projectId: string
|
||||
documentId: string
|
||||
collectionName: string
|
||||
fileId: string
|
||||
storageKey?: string | undefined
|
||||
}
|
||||
},
|
||||
ctx: EventContext,
|
||||
): Promise<void> {
|
||||
const scope = await projectScopeOf(ctx, event.payload)
|
||||
await ctx.container.resolve(JOB_QUEUE).enqueue(
|
||||
scope,
|
||||
REMOVE_JOB,
|
||||
{
|
||||
documentId: event.payload.documentId,
|
||||
collectionName: event.payload.collectionName,
|
||||
fileId: event.payload.fileId,
|
||||
...(event.payload.storageKey === undefined ? {} : { storageKey: event.payload.storageKey }),
|
||||
},
|
||||
{
|
||||
key: removeJobKey(event.payload.collectionName, event.payload.fileId),
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
async function onSourceDeleted(
|
||||
event: { payload: { orgId: string; workspaceId: string; projectId: string; fileId: string; storageKey: string } },
|
||||
ctx: EventContext,
|
||||
): Promise<void> {
|
||||
const scope = await projectScopeOf(ctx, event.payload)
|
||||
// The file's row is gone by now: the key it had travels with the removal (F407).
|
||||
await requestSourceRemoval(knowledgeDeps(ctx.container), scope, event.payload.fileId, event.payload.storageKey)
|
||||
}
|
||||
|
||||
export const KNOWLEDGE_JOBS = { [INDEX_JOB]: indexJob, [REMOVE_JOB]: removeJob, [AUTO_INDEX_JOB]: autoIndexJob }
|
||||
export const KNOWLEDGE_SUBSCRIPTIONS = [
|
||||
subscribe(fileDeleted, onSourceDeleted),
|
||||
subscribe(documentIndexRequested, onDocumentIndexRequested),
|
||||
subscribe(documentRemoveRequested, onDocumentRemoveRequested),
|
||||
]
|
||||
@@ -0,0 +1,280 @@
|
||||
// The managed search service: the platform's documents-index service through the kit-platform
|
||||
// Knowledge client. Indexing and search run server-side there (the service owns chunking,
|
||||
// embeddings and the vector store); this module supplies the job lifecycle and the citations it
|
||||
// answers with.
|
||||
//
|
||||
// Contract notes (the service's real shapes): the index submission carries the content's SHA-256
|
||||
// (computed here from the job's bytes) as its version assertion; a quota denial surfaces on the
|
||||
// job, which the poll below reads as the job's typed error. Blob pathnames and returned citations
|
||||
// resolve through the files module's scoped rows, so an opaque file id never becomes a pathname
|
||||
// and a foreign or stale service reference never becomes a valid app hit.
|
||||
|
||||
import { createHash } from "node:crypto"
|
||||
import { setTimeout as delay } from "node:timers/promises"
|
||||
import {
|
||||
type FilesDeps,
|
||||
findFileStorageByKey,
|
||||
getFileStorage,
|
||||
platformBlobPathname,
|
||||
platformBlobStorageKey,
|
||||
} from "@conusai-kit/files"
|
||||
import { KNOWLEDGE_LIMITS, type Knowledge, type KnowledgeClient, KnowledgeError } from "@conusai-kit/platform"
|
||||
import type { KnowledgeHit, KnowledgePassageIndex, KnowledgeScope, KnowledgeSearchService } from "../core/ports.js"
|
||||
import { platformKnowledgeReadiness } from "./readiness.js"
|
||||
import { permanentRefusal } from "./refusals.js"
|
||||
|
||||
/** The service's tenant partition of a project: a search of the project and its indexed documents agree on it. */
|
||||
const tenantOf = (scope: KnowledgeScope): string =>
|
||||
createHash("sha256")
|
||||
.update(JSON.stringify([scope.orgId, scope.workspaceId, scope.projectId]))
|
||||
.digest("hex")
|
||||
|
||||
/** A passage's reference: the file's own plus `#p{page}.{index}` (the automatic PDF intake, core-11). */
|
||||
const passageRef = (documentRef: string, passage: { page: number; index: number }): string =>
|
||||
`${documentRef}#p${passage.page}.${passage.index}`
|
||||
|
||||
const PASSAGE_REF = /^(.+)#p([1-9][0-9]*)\.(0|[1-9][0-9]*)$/
|
||||
|
||||
/**
|
||||
* `environment` is the deployment's (`CONUSAI_ENVIRONMENT`): the service keeps a collection in one environment,
|
||||
* and a deployment token reaches only its own.
|
||||
*/
|
||||
export function platformKnowledgeSearch(
|
||||
client: Knowledge,
|
||||
files: FilesDeps,
|
||||
options: { environment?: string | undefined; timeoutMs?: number; intervalMs?: number } = {},
|
||||
): KnowledgeSearchService {
|
||||
const environment = options.environment?.trim()
|
||||
const configuredEnvironment = (): string => {
|
||||
if (!environment)
|
||||
throw new KnowledgeError("configuration_missing", "CONUSAI_ENVIRONMENT is required for knowledge indexing")
|
||||
return environment
|
||||
}
|
||||
const unwrap = (error: unknown): never => {
|
||||
// The client's errors are already typed by reason; a non-knowledge failure is unavailable.
|
||||
if (error instanceof KnowledgeError) throw error
|
||||
throw new KnowledgeError(
|
||||
"runtime_unavailable",
|
||||
error instanceof Error ? error.message : "the knowledge service failed",
|
||||
true,
|
||||
)
|
||||
}
|
||||
/** The same, for an index job: a refusal waiting cannot fix is recorded on the document, not retried. */
|
||||
const unwrapJob = (error: unknown): never => {
|
||||
if (error instanceof KnowledgeError) {
|
||||
const permanent = permanentRefusal(error)
|
||||
if (permanent !== undefined) throw Object.assign(error, { permanent })
|
||||
}
|
||||
return unwrap(error)
|
||||
}
|
||||
const documentRef = async (scope: KnowledgeScope, fileId: string): Promise<string> => {
|
||||
const reference = await getFileStorage(files, scope, fileId)
|
||||
return platformBlobPathname(scope, reference.storageKey)
|
||||
}
|
||||
|
||||
return {
|
||||
readiness(scope) {
|
||||
if (!environment)
|
||||
return Promise.resolve({
|
||||
state: "unavailable",
|
||||
reasons: ["CONUSAI_ENVIRONMENT is not set"],
|
||||
refusal: "configuration_missing",
|
||||
})
|
||||
return platformKnowledgeReadiness(client).readiness(scope)
|
||||
},
|
||||
|
||||
async index(scope, input) {
|
||||
try {
|
||||
// The service answers 404 for a collection it does not hold. An app's collection is only a row of
|
||||
// this module (a write cannot leave the process), so the first index creates it; creating it again
|
||||
// finds it.
|
||||
const activeEnvironment = configuredEnvironment()
|
||||
const reference = await documentRef(scope, input.fileId)
|
||||
const signal = AbortSignal.any([
|
||||
AbortSignal.timeout(options.timeoutMs ?? 300_000),
|
||||
...(input.signal ? [input.signal] : []),
|
||||
])
|
||||
signal.throwIfAborted()
|
||||
await client.createCollection({ collection: input.collectionName, environment: activeEnvironment }, { signal })
|
||||
const { jobId } = await client.index(
|
||||
{
|
||||
collection: input.collectionName,
|
||||
documentRef: reference,
|
||||
options: {
|
||||
visibility: "tenant",
|
||||
tenant: tenantOf(scope),
|
||||
contentSha256: createHash("sha256").update(input.bytes).digest("hex"),
|
||||
},
|
||||
},
|
||||
{ signal },
|
||||
)
|
||||
// The service's job is durable; poll until it leaves the queue. A caller-side cancel is
|
||||
// the job's own AbortSignal, checked between polls.
|
||||
for (;;) {
|
||||
signal.throwIfAborted()
|
||||
const job = await client.job(jobId, { signal })
|
||||
if (job.status === "succeeded") return
|
||||
if (job.status === "failed")
|
||||
throw new KnowledgeError(
|
||||
job.error === "quota-exhausted" ? "quota_exhausted" : "runtime_unavailable",
|
||||
job.error ?? "the indexing job failed",
|
||||
)
|
||||
if (job.status === "cancelled")
|
||||
throw new KnowledgeError("runtime_unavailable", "the indexing job was cancelled")
|
||||
await delay(options.intervalMs ?? 1_000, undefined, { signal })
|
||||
}
|
||||
} catch (error) {
|
||||
return unwrapJob(error)
|
||||
}
|
||||
},
|
||||
|
||||
async search(scope, input) {
|
||||
try {
|
||||
configuredEnvironment()
|
||||
const results = await client.search({
|
||||
collection: input.collectionName,
|
||||
query: input.query,
|
||||
limit: input.limit,
|
||||
tenant: tenantOf(scope),
|
||||
})
|
||||
const hits: KnowledgeHit[] = []
|
||||
for (const [at, hit] of results.entries()) {
|
||||
// A passage of a PDF the intake indexed names its file's reference, and cites from its metadata:
|
||||
// the service reports pages only for sources it parsed itself.
|
||||
const passage = PASSAGE_REF.exec(hit.documentRef)
|
||||
const storageKey = platformBlobStorageKey(scope, passage?.[1] ?? hit.documentRef)
|
||||
if (storageKey === null) continue
|
||||
const reference = await findFileStorageByKey(files, scope, storageKey)
|
||||
if (reference === null) continue
|
||||
const metadata = "metadata" in hit ? (hit.metadata as Readonly<Record<string, string>>) : {}
|
||||
if (passage && metadata.file_id !== reference.file.fileId) continue
|
||||
const page = passage ? Number(passage[2]) : hit.citations[0]?.page
|
||||
const blockId = passage ? metadata.block_ids?.split(",")[0] || undefined : hit.citations[0]?.blockId
|
||||
hits.push({
|
||||
fileId: reference.file.fileId,
|
||||
filename: reference.file.filename,
|
||||
collectionName: input.collectionName,
|
||||
score: hit.score,
|
||||
chunk: at + 1,
|
||||
snippet: hit.snippet,
|
||||
...(page === undefined ? {} : { page }),
|
||||
...(blockId === undefined ? {} : { blockId }),
|
||||
...("chunkId" in hit && typeof hit.chunkId === "string" ? { chunkId: hit.chunkId } : {}),
|
||||
})
|
||||
}
|
||||
return hits
|
||||
} catch (error) {
|
||||
// Nothing has been indexed into this collection yet, so the service has not heard of it.
|
||||
if (error instanceof KnowledgeError && error.reason === "unavailable_resource") return []
|
||||
return unwrap(error)
|
||||
}
|
||||
},
|
||||
|
||||
async remove(scope, input) {
|
||||
try {
|
||||
// Every removal asked for since F407 names its object; only a record from before it is looked up.
|
||||
const reference =
|
||||
input.storageKey === undefined
|
||||
? await documentRef(scope, input.fileId)
|
||||
: platformBlobPathname(scope, input.storageKey)
|
||||
await client.remove(reference)
|
||||
} catch (error) {
|
||||
return unwrap(error)
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The automatic PDF intake's index (core-11): one inline item per passage, in the project's tenant partition,
|
||||
* under the file's Blob reference plus `#p{page}.{index}`, with its page, its blocks and its file in metadata.
|
||||
* Items go at most the service's count per call, and every item's durable job is waited for; a refusal waiting
|
||||
* cannot fix is the document's, as with a whole file.
|
||||
*/
|
||||
export function platformPassageIndex(
|
||||
client: Pick<KnowledgeClient, "createCollection" | "indexInline" | "job" | "remove">,
|
||||
files: FilesDeps,
|
||||
options: { environment?: string | undefined; timeoutMs?: number; intervalMs?: number } = {},
|
||||
): KnowledgePassageIndex {
|
||||
const environment = options.environment?.trim()
|
||||
const failure = (error: unknown): never => {
|
||||
if (error instanceof KnowledgeError) {
|
||||
const permanent = permanentRefusal(error)
|
||||
if (permanent !== undefined) throw Object.assign(error, { permanent })
|
||||
throw error
|
||||
}
|
||||
throw new KnowledgeError(
|
||||
"runtime_unavailable",
|
||||
error instanceof Error ? error.message : "the knowledge service failed",
|
||||
true,
|
||||
)
|
||||
}
|
||||
|
||||
return {
|
||||
async reference(scope, fileId) {
|
||||
const stored = await getFileStorage(files, scope, fileId)
|
||||
return platformBlobPathname(scope, stored.storageKey)
|
||||
},
|
||||
|
||||
async index(scope, input) {
|
||||
try {
|
||||
if (!environment)
|
||||
throw new KnowledgeError("configuration_missing", "CONUSAI_ENVIRONMENT is required for knowledge indexing")
|
||||
const signal = AbortSignal.any([
|
||||
AbortSignal.timeout(options.timeoutMs ?? 300_000),
|
||||
...(input.signal ? [input.signal] : []),
|
||||
])
|
||||
await client.createCollection({ collection: input.collectionName, environment }, { signal })
|
||||
const tenant = tenantOf(scope)
|
||||
const jobs: string[] = []
|
||||
for (let at = 0; at < input.passages.length; at += KNOWLEDGE_LIMITS.inlineItems) {
|
||||
signal.throwIfAborted()
|
||||
const accepted = await client.indexInline({
|
||||
collection: input.collectionName,
|
||||
items: input.passages.slice(at, at + KNOWLEDGE_LIMITS.inlineItems).map((passage) => ({
|
||||
documentRef: passageRef(input.documentRef, passage),
|
||||
text: passage.text,
|
||||
visibility: "tenant" as const,
|
||||
tenant,
|
||||
metadata: {
|
||||
page: String(passage.page),
|
||||
...(passage.blockIds.length === 0 ? {} : { block_ids: passage.blockIds.join(",") }),
|
||||
file_id: input.fileId,
|
||||
},
|
||||
})),
|
||||
})
|
||||
jobs.push(...accepted.map((job) => job.jobId))
|
||||
}
|
||||
for (const jobId of jobs) {
|
||||
for (;;) {
|
||||
signal.throwIfAborted()
|
||||
const job = await client.job(jobId, { signal })
|
||||
if (job.status === "succeeded") break
|
||||
if (job.status === "failed")
|
||||
throw new KnowledgeError(
|
||||
job.error === "quota-exhausted" ? "quota_exhausted" : "runtime_unavailable",
|
||||
job.error ?? "the indexing job failed",
|
||||
)
|
||||
if (job.status === "cancelled")
|
||||
throw new KnowledgeError("runtime_unavailable", "the indexing job was cancelled")
|
||||
await delay(options.intervalMs ?? 1_000, undefined, { signal })
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
return failure(error)
|
||||
}
|
||||
},
|
||||
|
||||
async remove(_scope, input) {
|
||||
for (const passage of input.passages) {
|
||||
try {
|
||||
await client.remove(passageRef(input.documentRef, passage))
|
||||
} catch (error) {
|
||||
// A passage the service never indexed (the run was refused before it) is already removed.
|
||||
if (error instanceof KnowledgeError && error.reason === "unavailable_resource") continue
|
||||
failure(error)
|
||||
}
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,154 @@
|
||||
// The handlers of the knowledge module's procedures: each resolves its ports from the request's
|
||||
// container and calls one use case. No rule lives here.
|
||||
|
||||
import type { Container } from "@conusai-kit/core"
|
||||
import { implement } from "@conusai-kit/core"
|
||||
import { DATA_SOURCE } from "@conusai-kit/data"
|
||||
import { filesDeps } from "@conusai-kit/files"
|
||||
import {
|
||||
autoIndexGet,
|
||||
autoIndexSet,
|
||||
collectionCreate,
|
||||
collectionList,
|
||||
documentGetMany,
|
||||
documentIndex,
|
||||
documentList,
|
||||
documentRemove,
|
||||
knowledgeSearch,
|
||||
knowledgeStatus,
|
||||
settingsDisableDocumentAi,
|
||||
settingsEnableDocumentAi,
|
||||
settingsGet,
|
||||
} from "../contract/index.js"
|
||||
import { getAutoIndex, setAutoIndex } from "../core/auto-index.js"
|
||||
import { createCollection, listCollections } from "../core/collections.js"
|
||||
import { getManyDocuments, listDocuments, requestIndex, requestRemove } from "../core/documents.js"
|
||||
import {
|
||||
flagsOf,
|
||||
KNOWLEDGE_ENABLEMENT,
|
||||
KNOWLEDGE_PASSAGES,
|
||||
KNOWLEDGE_READER,
|
||||
KNOWLEDGE_READINESS,
|
||||
KNOWLEDGE_SEARCH,
|
||||
type KnowledgeDeps,
|
||||
} from "../core/ports.js"
|
||||
import { searchKnowledge } from "../core/search.js"
|
||||
import { disableDocumentAi, enableDocumentAi, getSettings, type SettingsDeps } from "../core/settings.js"
|
||||
import { type KnowledgeStatusDeps, knowledgeStatusOf } from "../core/status.js"
|
||||
import { knowledgeRefusal } from "./refusals.js"
|
||||
|
||||
/** The ports a write or read handler needs. Never the search service: that port is an effect, and
|
||||
* resolving it from either is a kind violation. */
|
||||
export function knowledgeDeps(container: Container): KnowledgeDeps {
|
||||
const data = container.resolve(DATA_SOURCE)
|
||||
return { data, files: filesDeps(container), flags: flagsOf(container, data) }
|
||||
}
|
||||
|
||||
/**
|
||||
* The effect handler's deps: the search service, bound in the processes that may leave the process, and the
|
||||
* PDF intake's reader, passage index and enablement checks where the worker binds the reader (the jobs; a
|
||||
* request never reads a PDF).
|
||||
*/
|
||||
export function knowledgeSearchDeps(container: Container): KnowledgeDeps {
|
||||
return {
|
||||
...knowledgeDeps(container),
|
||||
search: container.resolve(KNOWLEDGE_SEARCH),
|
||||
...(container.has(KNOWLEDGE_READER) ? { reader: container.resolve(KNOWLEDGE_READER) } : {}),
|
||||
...(container.has(KNOWLEDGE_PASSAGES) ? { passages: container.resolve(KNOWLEDGE_PASSAGES) } : {}),
|
||||
...(container.has(KNOWLEDGE_READER) ? { enablement: container.resolve(KNOWLEDGE_ENABLEMENT) } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
/** The status read's deps: the backend's readiness, which a read may resolve, instead of the search service. */
|
||||
function knowledgeStatusDeps(container: Container): KnowledgeStatusDeps {
|
||||
const data = container.resolve(DATA_SOURCE)
|
||||
return {
|
||||
data,
|
||||
flags: flagsOf(container, data),
|
||||
...(container.has(KNOWLEDGE_READINESS) ? { readiness: container.resolve(KNOWLEDGE_READINESS) } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
/** The enablement flow's deps, for the effect that leaves the process. The other settings procedures read and write the module's own row. */
|
||||
function settingsDeps(container: Container): SettingsDeps {
|
||||
return { data: container.resolve(DATA_SOURCE), platform: container.resolve(KNOWLEDGE_ENABLEMENT) }
|
||||
}
|
||||
|
||||
const settingsDataOf = (container: Container) => ({ data: container.resolve(DATA_SOURCE) })
|
||||
|
||||
export const collectionListImpl = implement(collectionList, ({ ctx }) =>
|
||||
listCollections(knowledgeDeps(ctx.container), ctx.scope),
|
||||
)
|
||||
|
||||
export const collectionCreateImpl = implement(collectionCreate, ({ input, ctx }) =>
|
||||
createCollection(knowledgeDeps(ctx.container), ctx.scope, input.name),
|
||||
)
|
||||
|
||||
export const documentIndexImpl = implement(documentIndex, ({ input, ctx }) =>
|
||||
requestIndex(knowledgeDeps(ctx.container), ctx.scope, input),
|
||||
)
|
||||
|
||||
export const documentListImpl = implement(documentList, ({ input, ctx }) =>
|
||||
listDocuments(knowledgeDeps(ctx.container), ctx.scope, input.collectionName),
|
||||
)
|
||||
|
||||
export const documentGetManyImpl = implement(documentGetMany, async ({ input, ctx }) => ({
|
||||
items: await getManyDocuments(knowledgeDeps(ctx.container), ctx.scope, input.ids),
|
||||
}))
|
||||
|
||||
export const documentRemoveImpl = implement(documentRemove, ({ input, ctx }) =>
|
||||
requestRemove(knowledgeDeps(ctx.container), ctx.scope, input),
|
||||
)
|
||||
|
||||
export const knowledgeSearchImpl = implement(knowledgeSearch, async ({ input, ctx }) => {
|
||||
try {
|
||||
return await searchKnowledge(knowledgeSearchDeps(ctx.container), ctx.scope, {
|
||||
collectionName: input.collectionName,
|
||||
query: input.query,
|
||||
...(input.limit === undefined ? {} : { limit: input.limit }),
|
||||
})
|
||||
} catch (error) {
|
||||
throw knowledgeRefusal(error)
|
||||
}
|
||||
})
|
||||
|
||||
export const knowledgeStatusImpl = implement(knowledgeStatus, ({ ctx }) =>
|
||||
knowledgeStatusOf(knowledgeStatusDeps(ctx.container), ctx.scope),
|
||||
)
|
||||
|
||||
export const settingsGetImpl = implement(settingsGet, ({ ctx }) =>
|
||||
getSettings(settingsDataOf(ctx.container), ctx.scope),
|
||||
)
|
||||
|
||||
export const settingsEnableDocumentAiImpl = implement(settingsEnableDocumentAi, async ({ ctx }) => {
|
||||
const settings = await enableDocumentAi(settingsDeps(ctx.container), ctx.scope)
|
||||
return { enabled: settings.enabled, backend: settings.backend ?? "platform" }
|
||||
})
|
||||
|
||||
export const settingsDisableDocumentAiImpl = implement(settingsDisableDocumentAi, ({ ctx }) =>
|
||||
disableDocumentAi(settingsDataOf(ctx.container), ctx.scope),
|
||||
)
|
||||
|
||||
export const autoIndexGetImpl = implement(autoIndexGet, ({ ctx }) =>
|
||||
getAutoIndex(settingsDataOf(ctx.container), ctx.scope),
|
||||
)
|
||||
|
||||
export const autoIndexSetImpl = implement(autoIndexSet, ({ input, ctx }) =>
|
||||
setAutoIndex(knowledgeDeps(ctx.container), ctx.scope, input.collectionName),
|
||||
)
|
||||
|
||||
export const KNOWLEDGE_PROCEDURE_IMPLS = [
|
||||
collectionListImpl,
|
||||
collectionCreateImpl,
|
||||
documentIndexImpl,
|
||||
documentListImpl,
|
||||
documentGetManyImpl,
|
||||
documentRemoveImpl,
|
||||
knowledgeSearchImpl,
|
||||
knowledgeStatusImpl,
|
||||
settingsGetImpl,
|
||||
settingsEnableDocumentAiImpl,
|
||||
settingsDisableDocumentAiImpl,
|
||||
autoIndexGetImpl,
|
||||
autoIndexSetImpl,
|
||||
]
|
||||
@@ -0,0 +1,46 @@
|
||||
// The automatic PDF intake's reader (core-11): the platform's document reader (`DocumentReader`, kit-platform's
|
||||
// `@conusai-sdk/ocr` wired to the deployment contract) and the read document's passages
|
||||
// (`@conusai-sdk/ocr/passages`), cut to the knowledge service's limit for one indexed item. The one thing this
|
||||
// adapter decides is what is permanent: a job the platform failed, or a document whose blocks no metadata
|
||||
// value can cite (`passages` throws `RangeError`), is not indexable; anything else is thrown for the queue.
|
||||
|
||||
import type { DocumentReader } from "@conusai-kit/platform"
|
||||
import { KNOWLEDGE_LIMITS } from "@conusai-kit/platform"
|
||||
import { OcrError } from "@conusai-sdk/ocr"
|
||||
import { passages } from "@conusai-sdk/ocr/passages"
|
||||
import type { KnowledgeReader } from "../core/ports.js"
|
||||
|
||||
export interface SdkKnowledgeReaderOptions {
|
||||
/** Poll interval of `wait`, in milliseconds. */
|
||||
readonly intervalMs?: number
|
||||
/** Give up on one job after this long; the queue's retry submits again under the same key. */
|
||||
readonly timeoutMs?: number
|
||||
}
|
||||
|
||||
export function sdkKnowledgeReader(
|
||||
reader: Pick<DocumentReader, "submit" | "wait">,
|
||||
options: SdkKnowledgeReaderOptions = {},
|
||||
): KnowledgeReader {
|
||||
return {
|
||||
async read(bytes, { filename, idempotencyKey, signal }) {
|
||||
try {
|
||||
const job = await reader.submit(bytes, { filename, idempotencyKey })
|
||||
const document = await reader.wait(job.id, {
|
||||
...(options.intervalMs === undefined ? {} : { intervalMs: options.intervalMs }),
|
||||
...(options.timeoutMs === undefined ? {} : { timeoutMs: options.timeoutMs }),
|
||||
...(signal ? { signal } : {}),
|
||||
})
|
||||
return passages(document, { maxBytes: KNOWLEDGE_LIMITS.inlineTextBytes }).map((passage) => ({
|
||||
page: passage.page,
|
||||
index: passage.index,
|
||||
blockIds: passage.blockIds,
|
||||
text: passage.text,
|
||||
}))
|
||||
} catch (error) {
|
||||
if ((error instanceof OcrError && error.type === "job-failed") || error instanceof RangeError)
|
||||
throw Object.assign(error, { permanent: "not_indexable" })
|
||||
throw error
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
// The platform service's readiness, on its own so the status read can ask it without the search service
|
||||
// (`KNOWLEDGE_READINESS`): the search adapter answers the same way, from here.
|
||||
|
||||
import { type Knowledge, KnowledgeError } from "@conusai-kit/platform"
|
||||
import type { KnowledgeReadiness } from "../core/ports.js"
|
||||
import { refusalOfReason, stateOfRefusal } from "./refusals.js"
|
||||
|
||||
/**
|
||||
* The platform's managed service answers for itself. A refusal is the service's answer, as a typed code
|
||||
* the page shows in the reader's language; a failure to ask it is its being unavailable.
|
||||
*/
|
||||
export const platformKnowledgeReadiness = (client: Pick<Knowledge, "readiness">): KnowledgeReadiness => ({
|
||||
async readiness() {
|
||||
try {
|
||||
const answer = await client.readiness()
|
||||
return { state: answer.state, reasons: [...answer.reasons] }
|
||||
} catch (error) {
|
||||
if (error instanceof KnowledgeError) {
|
||||
const refusal = refusalOfReason(error.reason)
|
||||
return { state: stateOfRefusal(refusal), reasons: [], ...(refusal === undefined ? {} : { refusal }) }
|
||||
}
|
||||
return { state: "unavailable", reasons: [], refusal: "runtime_unavailable" }
|
||||
}
|
||||
},
|
||||
})
|
||||
@@ -0,0 +1,77 @@
|
||||
// A refusal of the platform's knowledge service as this module's one closed set of codes
|
||||
// (`KNOWLEDGE_REFUSALS`), which the pages and forms show in the reader's language. A refusal that is the
|
||||
// deployment's or the platform's says so; the service's own words stay in `detail`, for the log and the API.
|
||||
|
||||
import { AppError, notFound } from "@conusai-kit/core"
|
||||
import { KnowledgeError, type KnowledgeReasonCode } from "@conusai-kit/platform"
|
||||
import type { KnowledgeRefusal, ServiceState } from "../contract/index.js"
|
||||
|
||||
const REFUSAL: Readonly<Record<Exclude<KnowledgeReasonCode, "unavailable_resource">, KnowledgeRefusal>> = {
|
||||
plan_required: "plan_required",
|
||||
feature_disabled: "feature_disabled",
|
||||
quota_exhausted: "quota_exhausted",
|
||||
budget_exhausted: "budget_exhausted",
|
||||
// The service refused the token itself: `knowledge.collections:use` is missing, `*` never carries it.
|
||||
forbidden: "scope_missing",
|
||||
indexing: "indexing",
|
||||
approval_required: "approval_required",
|
||||
runtime_unavailable: "runtime_unavailable",
|
||||
configuration_missing: "configuration_missing",
|
||||
// The platform refused the token's identity: nothing the person can act on, the app is not served.
|
||||
unauthenticated: "runtime_unavailable",
|
||||
}
|
||||
|
||||
const STATUS: Readonly<Record<KnowledgeRefusal, number>> = {
|
||||
plan_required: 403,
|
||||
feature_disabled: 403,
|
||||
approval_required: 403,
|
||||
quota_exhausted: 429,
|
||||
budget_exhausted: 429,
|
||||
scope_missing: 503,
|
||||
indexing: 503,
|
||||
runtime_unavailable: 503,
|
||||
configuration_missing: 503,
|
||||
}
|
||||
|
||||
const TITLE: Readonly<Record<KnowledgeRefusal, string>> = {
|
||||
plan_required: "The organization's plan does not include document search",
|
||||
feature_disabled: "Document search is disabled by policy",
|
||||
approval_required: "The platform asks for an approval first",
|
||||
quota_exhausted: "The indexing quota is exhausted",
|
||||
budget_exhausted: "The AI budget is exhausted",
|
||||
scope_missing: "This deployment's token does not hold knowledge.collections:use",
|
||||
indexing: "The knowledge service is building its index",
|
||||
runtime_unavailable: "The knowledge service is unavailable",
|
||||
configuration_missing: "The knowledge deployment configuration is incomplete",
|
||||
}
|
||||
|
||||
/** The typed refusal of a service error; a resource it does not hold has none, it is simply not there. */
|
||||
export function refusalOfReason(reason: KnowledgeReasonCode): KnowledgeRefusal | undefined {
|
||||
return reason === "unavailable_resource" ? undefined : REFUSAL[reason]
|
||||
}
|
||||
|
||||
/** The readiness state a refusal puts the service in: a plan or a policy is "disabled", not "down". */
|
||||
export function stateOfRefusal(refusal: KnowledgeRefusal | undefined): ServiceState {
|
||||
if (refusal === "plan_required" || refusal === "feature_disabled") return "disabled"
|
||||
return refusal === "indexing" ? "indexing" : "unavailable"
|
||||
}
|
||||
|
||||
/** The app error for anything the service threw; a programming error passes through as it is. */
|
||||
export function knowledgeRefusal(error: unknown): unknown {
|
||||
if (!(error instanceof KnowledgeError)) return error
|
||||
const refusal = refusalOfReason(error.reason)
|
||||
if (refusal === undefined) return notFound(error.message)
|
||||
return new AppError({
|
||||
type: `/problems/${refusal}`,
|
||||
title: TITLE[refusal],
|
||||
status: STATUS[refusal],
|
||||
code: refusal,
|
||||
detail: error.message,
|
||||
})
|
||||
}
|
||||
|
||||
/** A refusal waiting cannot fix: the index job records it on the document instead of retrying it. */
|
||||
export function permanentRefusal(error: KnowledgeError): KnowledgeRefusal | undefined {
|
||||
const refusal = refusalOfReason(error.reason)
|
||||
return refusal === "indexing" || refusal === "runtime_unavailable" ? undefined : refusal
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
// The live collection of the knowledge module (umbrella TS6.1). The live server announces a change of
|
||||
// `knowledge_document` -- an index asked for, a job taking it, the platform answering indexed or failed, a
|
||||
// removal -- as `{ entity: "knowledge.document", id }`, ids only; a reader reads the document back through
|
||||
// `knowledge.document.getMany`, so the procedure's permission and the repository's tenant wall decide what a
|
||||
// member sees. A topic is a project, so a page that shows one collection filters the rows by its name.
|
||||
// `pnpm kit check` refuses a collection without its batch-get procedure.
|
||||
|
||||
import type { ModuleCollection } from "@conusai-kit/core"
|
||||
|
||||
export const DOCUMENT_COLLECTION = {
|
||||
entity: "knowledge.document",
|
||||
table: "knowledge_document",
|
||||
key: "documentId",
|
||||
scopeLevel: "project",
|
||||
read: "knowledge.knowledge.search",
|
||||
list: "knowledge.document.list",
|
||||
getMany: "knowledge.document.getMany",
|
||||
} as const satisfies ModuleCollection
|
||||
|
||||
export const KNOWLEDGE_COLLECTIONS: readonly ModuleCollection[] = [DOCUMENT_COLLECTION]
|
||||
@@ -0,0 +1,50 @@
|
||||
// The constants of the knowledge module: its feature flag, jobs, events, audit actions and the
|
||||
// indexable content types.
|
||||
|
||||
/** The module flag: knowledge retrieval and indexing, off until an administrator enables document AI. */
|
||||
export const KNOWLEDGE_FLAG = "knowledge.enabled"
|
||||
|
||||
export const KNOWLEDGE_FLAG_DEFAULTS = {
|
||||
[KNOWLEDGE_FLAG]: false,
|
||||
} as const
|
||||
|
||||
/** The worker job that hands one document to the platform's knowledge service. */
|
||||
export const INDEX_JOB = "knowledge.index"
|
||||
|
||||
/** The worker job that removes one document from the platform's knowledge service. */
|
||||
export const REMOVE_JOB = "knowledge.remove"
|
||||
|
||||
export const COLLECTION_CREATED = "knowledge.collection.created"
|
||||
export const DOCUMENT_INDEX_REQUESTED = "knowledge.document.index_requested"
|
||||
export const DOCUMENT_REMOVE_REQUESTED = "knowledge.document.remove_requested"
|
||||
export const DOCUMENT_AI_ENABLED = "knowledge.settings.document_ai_enabled"
|
||||
export const DOCUMENT_AI_DISABLED = "knowledge.settings.document_ai_disabled"
|
||||
/** A project's automatic PDF indexing switched on (naming a collection) or off. */
|
||||
export const AUTO_INDEX_CHANGED = "knowledge.autoindex.changed"
|
||||
|
||||
/** The worker job that reads one uploaded PDF and indexes its passages into the project's opted-in collection. */
|
||||
export const AUTO_INDEX_JOB = "knowledge.autoindex"
|
||||
|
||||
/** The files processor that enqueues it on every upload of a matching type; also its jobs' singleton namespace. */
|
||||
export const AUTO_INDEX_PROCESSOR_KEY = "knowledge.autoindex"
|
||||
|
||||
/** What the automatic indexing reads, through the platform's document reader. */
|
||||
export const AUTO_INDEX_CONTENT_TYPES = ["application/pdf"] as const
|
||||
|
||||
/** A PDF's ceiling for the reader: the files module's upload ceiling, as the documents module reads one. */
|
||||
export const MAX_AUTO_INDEX_BYTES = 50 * 1024 * 1024
|
||||
|
||||
/**
|
||||
* What the indexer reads as text. PDFs and scans are the documents module's worked example; this
|
||||
* module indexes text-shaped files, and refuses anything else at the door with a named field.
|
||||
*/
|
||||
export const INDEXABLE_CONTENT_TYPES = ["text/plain", "text/markdown", "application/json", "text/csv"] as const
|
||||
|
||||
/** A search query's ceiling; a retrieval question is a sentence, not a document. */
|
||||
export const MAX_QUERY_LENGTH = 500
|
||||
|
||||
/** What one search returns at most, however much the caller asks (the retrieval limit contract). */
|
||||
export const MAX_SEARCH_RESULTS = 25
|
||||
|
||||
/** An indexed file's ceiling: one megabyte of text is already far past useful chunking. */
|
||||
export const MAX_INDEX_BYTES = 1_048_576
|
||||
@@ -0,0 +1,105 @@
|
||||
// The tables of the knowledge module. Five.
|
||||
//
|
||||
// `knowledge_collection` is one named collection of documents, project-owned; the collection name
|
||||
// is the slug procedures take (a procedure's input carries slugs, never ids). `knowledge_document`
|
||||
// is one file's indexing state in one collection — the row the UI's progress states read, and the
|
||||
// idempotency key of the index job (a file is indexed into a collection once, and re-indexing the
|
||||
// same pair resumes the one row). `knowledge_config` is the organization's enablement state: the
|
||||
// durable answer to "is document AI on" and which backend serves it. `knowledge_auto_index` is a
|
||||
// project's opt-in to indexing its uploaded PDFs, and `knowledge_passage_set` what one such PDF sent
|
||||
// to the service, so removing it can tombstone each passage. No secret ever lands here.
|
||||
|
||||
import { defineEntity, idSchema, instantSchema } from "@conusai-kit/data"
|
||||
import { z } from "zod"
|
||||
import { DOCUMENT_STATUSES, KNOWLEDGE_BACKENDS } from "./statuses.js"
|
||||
|
||||
export const collections = defineEntity({
|
||||
name: "collection",
|
||||
table: "knowledge_collection",
|
||||
tenancy: "project",
|
||||
fields: {
|
||||
name: z.string().min(1).max(100),
|
||||
createdBy: z.string().min(1).max(128),
|
||||
},
|
||||
unique: [["name"]],
|
||||
})
|
||||
|
||||
export const documents = defineEntity({
|
||||
name: "document",
|
||||
table: "knowledge_document",
|
||||
tenancy: "project",
|
||||
fields: {
|
||||
collectionName: z.string().min(1).max(100),
|
||||
/** The row of the files module this document indexes: an opaque id, never a foreign key. */
|
||||
fileId: idSchema,
|
||||
filename: z.string().min(1).max(255),
|
||||
status: z.enum(DOCUMENT_STATUSES),
|
||||
/** A stable failure code when the indexing failed; null otherwise. */
|
||||
error: z.string().max(200).nullable(),
|
||||
indexedAt: z.string().nullable(),
|
||||
requestedBy: z.string().min(1).max(128),
|
||||
},
|
||||
// One row per (collection, file): a re-index resumes the same row instead of duplicating it.
|
||||
unique: [["collectionName", "fileId"]],
|
||||
indexes: [["collectionName"]],
|
||||
})
|
||||
|
||||
export const config = defineEntity({
|
||||
name: "config",
|
||||
table: "knowledge_config",
|
||||
tenancy: "org",
|
||||
fields: {
|
||||
enabled: z.boolean(),
|
||||
/** Which search backend serves this organization. */
|
||||
backend: z.enum(KNOWLEDGE_BACKENDS),
|
||||
// The three below held the app's own vector store, which is gone (ADR-061); the migration that made
|
||||
// them has run, so they stay as "managed, nothing to link" and nothing reads them.
|
||||
source: z.enum(["managed"]),
|
||||
serviceId: z.string().max(128).nullable(),
|
||||
url: z.string().max(500).nullable(),
|
||||
updatedBy: z.string().min(1).max(128),
|
||||
/** When the configuration last changed; the managed updated_at is the row's own bookkeeping. */
|
||||
configuredAt: instantSchema,
|
||||
},
|
||||
})
|
||||
|
||||
/**
|
||||
* A project's opt-in to indexing its uploaded PDFs automatically (core-11): no row means off. It names
|
||||
* the collection the files go into and the administrator who turned it on, whose permission every
|
||||
* indexing job checks again, since the job runs for nobody else.
|
||||
*/
|
||||
export const autoIndex = defineEntity({
|
||||
name: "autoindex",
|
||||
table: "knowledge_auto_index",
|
||||
tenancy: "project",
|
||||
fields: {
|
||||
collectionName: z.string().min(1).max(100),
|
||||
enabledBy: z.string().min(1).max(128),
|
||||
enabledAt: instantSchema,
|
||||
},
|
||||
})
|
||||
|
||||
/**
|
||||
* The passages an automatic PDF index sent for one document (core-11). The service indexed one inline
|
||||
* item per passage under the file's reference plus `#p{page}.{index}`, and removes a reference only
|
||||
* exactly, so the row keeps the file's reference (which outlives the file) and the count of passages
|
||||
* on each page, comma-separated from page 1 ("2,0,1": two on page 1, none on page 2, one on page 3).
|
||||
*/
|
||||
export const passageSets = defineEntity({
|
||||
name: "passageset",
|
||||
table: "knowledge_passage_set",
|
||||
tenancy: "project",
|
||||
fields: {
|
||||
collectionName: z.string().min(1).max(100),
|
||||
fileId: idSchema,
|
||||
documentRef: z.string().min(1).max(1024),
|
||||
pages: z.string().max(65_536),
|
||||
},
|
||||
unique: [["collectionName", "fileId"]],
|
||||
})
|
||||
|
||||
/** The tables of the first migration, as first shipped. */
|
||||
export const KNOWLEDGE_ENTITIES_V1 = Object.freeze([collections, documents, config])
|
||||
|
||||
/** Every table this module owns, in the order its migrations create them. */
|
||||
export const KNOWLEDGE_ENTITIES = Object.freeze([...KNOWLEDGE_ENTITIES_V1, autoIndex, passageSets])
|
||||
@@ -0,0 +1,41 @@
|
||||
// The events of the knowledge module. A write procedure publishes the request; the module's own
|
||||
// subscriber (running in the worker's outbox relay) enqueues the job, which is how a web request
|
||||
// never waits on the platform's knowledge service.
|
||||
|
||||
import { defineEvent } from "@conusai-kit/core"
|
||||
import { z } from "zod"
|
||||
|
||||
const inProject = {
|
||||
orgId: z.string(),
|
||||
workspaceId: z.string(),
|
||||
projectId: z.string(),
|
||||
}
|
||||
|
||||
/** A document was asked to be indexed: the subscriber enqueues the index job. */
|
||||
export const documentIndexRequested = defineEvent(
|
||||
"knowledge.documentIndexRequested",
|
||||
z.object({
|
||||
...inProject,
|
||||
documentId: z.string(),
|
||||
collectionName: z.string(),
|
||||
fileId: z.string(),
|
||||
}),
|
||||
)
|
||||
|
||||
/**
|
||||
* A document was asked to be removed: the subscriber enqueues the remove job. `storageKey` names the object to
|
||||
* remove, resolved when the removal is asked for (from `FileDeleted`, or from the stored file a person removes),
|
||||
* because the file may be gone when the job runs. It is optional only for a record written before it existed.
|
||||
*/
|
||||
export const documentRemoveRequested = defineEvent(
|
||||
"knowledge.documentRemoveRequested",
|
||||
z.object({
|
||||
...inProject,
|
||||
documentId: z.string(),
|
||||
collectionName: z.string(),
|
||||
fileId: z.string(),
|
||||
storageKey: z.string().optional(),
|
||||
}),
|
||||
)
|
||||
|
||||
export const KNOWLEDGE_EVENTS = [documentIndexRequested, documentRemoveRequested]
|
||||
@@ -0,0 +1,66 @@
|
||||
// The public API of the knowledge module: what a page, another module and the OpenAPI document may
|
||||
// import. Nothing here reaches core/, adapters/ or a driver.
|
||||
|
||||
export { DOCUMENT_COLLECTION, KNOWLEDGE_COLLECTIONS } from "./collections.js"
|
||||
export {
|
||||
AUTO_INDEX_CHANGED,
|
||||
AUTO_INDEX_CONTENT_TYPES,
|
||||
AUTO_INDEX_JOB,
|
||||
AUTO_INDEX_PROCESSOR_KEY,
|
||||
COLLECTION_CREATED,
|
||||
DOCUMENT_AI_DISABLED,
|
||||
DOCUMENT_AI_ENABLED,
|
||||
DOCUMENT_INDEX_REQUESTED,
|
||||
DOCUMENT_REMOVE_REQUESTED,
|
||||
INDEX_JOB,
|
||||
INDEXABLE_CONTENT_TYPES,
|
||||
KNOWLEDGE_FLAG,
|
||||
KNOWLEDGE_FLAG_DEFAULTS,
|
||||
MAX_AUTO_INDEX_BYTES,
|
||||
MAX_INDEX_BYTES,
|
||||
MAX_QUERY_LENGTH,
|
||||
MAX_SEARCH_RESULTS,
|
||||
REMOVE_JOB,
|
||||
} from "./constants.js"
|
||||
export {
|
||||
autoIndex,
|
||||
collections,
|
||||
config,
|
||||
documents,
|
||||
KNOWLEDGE_ENTITIES,
|
||||
KNOWLEDGE_ENTITIES_V1,
|
||||
passageSets,
|
||||
} from "./entities.js"
|
||||
export { documentIndexRequested, documentRemoveRequested, KNOWLEDGE_EVENTS } from "./events.js"
|
||||
export type { CollectionView, DocumentView, SearchResult } from "./procedures.js"
|
||||
export {
|
||||
autoIndexGet,
|
||||
autoIndexSet,
|
||||
collectionCreate,
|
||||
collectionList,
|
||||
collectionName,
|
||||
collectionViewSchema,
|
||||
documentGetMany,
|
||||
documentIndex,
|
||||
documentList,
|
||||
documentRemove,
|
||||
documentViewSchema,
|
||||
KNOWLEDGE_PROCEDURES,
|
||||
knowledgeSearch,
|
||||
knowledgeStatus,
|
||||
searchResultSchema,
|
||||
settingsDisableDocumentAi,
|
||||
settingsEnableDocumentAi,
|
||||
settingsGet,
|
||||
} from "./procedures.js"
|
||||
export type { DocumentStatus, EnablementReason, IndexFailure } from "./statuses.js"
|
||||
export {
|
||||
ENABLEMENT_REASONS,
|
||||
INDEX_FAILURES,
|
||||
KNOWLEDGE_BACKENDS,
|
||||
KNOWLEDGE_REFUSALS,
|
||||
type KnowledgeBackend,
|
||||
type KnowledgeRefusal,
|
||||
SERVICE_STATES,
|
||||
type ServiceState,
|
||||
} from "./statuses.js"
|
||||
@@ -0,0 +1,229 @@
|
||||
// The procedure contracts of the knowledge module: schemas and names, no handler and no port.
|
||||
//
|
||||
// The module is a retrieval layer over the app's documents: a collection of text files is indexed
|
||||
// (in the worker: the platform's knowledge service reads, chunks and embeds it), searched
|
||||
// with cited snippets, and managed through an explicit, idempotent enablement flow. The enablement
|
||||
// procedure is named enableDocumentAi per the Document AI enablement contract; its permission is the module's
|
||||
// manage permission, so the admin gate is one line in the manifest.
|
||||
|
||||
import { procedure } from "@conusai-kit/core"
|
||||
import { z } from "zod"
|
||||
import { AUTO_INDEX_CHANGED, COLLECTION_CREATED, DOCUMENT_AI_DISABLED, DOCUMENT_AI_ENABLED } from "./constants.js"
|
||||
import { type DocumentStatus, KNOWLEDGE_BACKENDS, KNOWLEDGE_REFUSALS, type ServiceState } from "./statuses.js"
|
||||
|
||||
const inProject = { org: z.string(), workspace: z.string(), project: z.string() }
|
||||
const inOrg = { org: z.string() }
|
||||
|
||||
export const collectionName = z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(100)
|
||||
.regex(/^[a-z][a-z0-9-]*$/, "lower-case letters, digits and dashes")
|
||||
|
||||
export const collectionViewSchema = z.object({
|
||||
name: collectionName,
|
||||
documentCount: z.number().int(),
|
||||
indexedCount: z.number().int(),
|
||||
createdBy: z.string(),
|
||||
})
|
||||
export type CollectionView = z.output<typeof collectionViewSchema>
|
||||
|
||||
export const documentViewSchema = z.object({
|
||||
documentId: z.string(),
|
||||
collectionName: collectionName,
|
||||
fileId: z.string(),
|
||||
filename: z.string(),
|
||||
status: z.enum(["queued", "indexing", "indexed", "failed", "cancelled"]),
|
||||
error: z.string().nullable(),
|
||||
indexedAt: z.string().nullable(),
|
||||
requestedBy: z.string(),
|
||||
})
|
||||
export type DocumentView = z.output<typeof documentViewSchema>
|
||||
|
||||
export const searchResultSchema = z.object({
|
||||
fileId: z.string(),
|
||||
filename: z.string(),
|
||||
collectionName: collectionName,
|
||||
snippet: z.string(),
|
||||
page: z.number().int().min(1).optional(),
|
||||
blockId: z.string().optional(),
|
||||
chunkId: z.string().optional(),
|
||||
/** 0..1; higher is closer. */
|
||||
score: z.number(),
|
||||
/** Result ordinal only. Page/block/chunkId carry actual returned source locators when available. */
|
||||
chunk: z.number().int().min(1),
|
||||
})
|
||||
export type SearchResult = z.output<typeof searchResultSchema>
|
||||
|
||||
export const collectionList = procedure("knowledge.collection.list")
|
||||
.kind("read")
|
||||
.scope("project")
|
||||
.permission("knowledge.knowledge.search")
|
||||
.input(z.object({ ...inProject }))
|
||||
.output(z.object({ collections: z.array(collectionViewSchema) }))
|
||||
.contract()
|
||||
|
||||
export const collectionCreate = procedure("knowledge.collection.create")
|
||||
.kind("write")
|
||||
.scope("project")
|
||||
.permission("knowledge.collection.manage")
|
||||
.input(z.object({ ...inProject, name: collectionName }))
|
||||
.output(collectionViewSchema)
|
||||
.audit(COLLECTION_CREATED)
|
||||
.contract()
|
||||
|
||||
export const documentIndex = procedure("knowledge.document.index")
|
||||
.kind("write")
|
||||
.scope("project")
|
||||
.permission("knowledge.knowledge.manage")
|
||||
.input(z.object({ ...inProject, collectionName: collectionName, fileId: z.string() }))
|
||||
.output(documentViewSchema)
|
||||
.contract()
|
||||
|
||||
export const documentList = procedure("knowledge.document.list")
|
||||
.kind("read")
|
||||
.scope("project")
|
||||
.permission("knowledge.knowledge.search")
|
||||
.input(z.object({ ...inProject, collectionName: collectionName }))
|
||||
.output(z.object({ documents: z.array(documentViewSchema) }))
|
||||
.contract()
|
||||
|
||||
/**
|
||||
* The live collection's batch-get (contract C-05): the documents with these ids, in whichever collection of
|
||||
* the project, under the list's permission. An id of another tenant, of another project or of nothing is
|
||||
* left out of the answer.
|
||||
*/
|
||||
export const documentGetMany = procedure("knowledge.document.getMany")
|
||||
.kind("read")
|
||||
.scope("project")
|
||||
.permission("knowledge.knowledge.search")
|
||||
.input(z.object({ ...inProject, ids: z.array(z.string().min(1).max(64)).min(1).max(100) }))
|
||||
.output(z.object({ items: z.array(documentViewSchema) }))
|
||||
.contract()
|
||||
|
||||
export const documentRemove = procedure("knowledge.document.remove")
|
||||
.kind("write")
|
||||
.scope("project")
|
||||
.permission("knowledge.knowledge.manage")
|
||||
.input(z.object({ ...inProject, collectionName: collectionName, fileId: z.string() }))
|
||||
.output(documentViewSchema)
|
||||
.contract()
|
||||
|
||||
/**
|
||||
* The retrieval call. It embeds the query through the AI gateway before it searches, so it is an
|
||||
* effect: a read procedure cannot leave the process. Exposed as an MCP tool on purpose (`.tool()`).
|
||||
*/
|
||||
export const knowledgeSearch = procedure("knowledge.search")
|
||||
.kind("effect")
|
||||
.scope("project")
|
||||
.permission("knowledge.knowledge.search")
|
||||
.tool()
|
||||
.input(
|
||||
z.object({
|
||||
...inProject,
|
||||
collectionName: collectionName,
|
||||
query: z.string().min(1).max(500),
|
||||
limit: z.number().int().min(1).max(25).optional(),
|
||||
}),
|
||||
)
|
||||
.output(z.object({ results: z.array(searchResultSchema) }))
|
||||
.contract()
|
||||
|
||||
export const knowledgeStatus = procedure("knowledge.status")
|
||||
.kind("read")
|
||||
.scope("project")
|
||||
.permission("knowledge.knowledge.search")
|
||||
.input(z.object({ ...inProject }))
|
||||
.output(
|
||||
z.object({
|
||||
/** The module flag's answer for this project. */
|
||||
enabled: z.boolean(),
|
||||
/** The service readiness, kept separate from the indexing indicator below. */
|
||||
service: z.object({
|
||||
state: z.enum(["registered", "indexing", "ready", "degraded", "disabled", "unavailable"]),
|
||||
reasons: z.array(z.string()),
|
||||
/** Why the service will not serve this deployment, when that is the answer: the page's own state. */
|
||||
refusal: z.enum(KNOWLEDGE_REFUSALS).optional(),
|
||||
}),
|
||||
documents: z.object({
|
||||
queued: z.number().int(),
|
||||
indexing: z.number().int(),
|
||||
indexed: z.number().int(),
|
||||
failed: z.number().int(),
|
||||
cancelled: z.number().int(),
|
||||
}),
|
||||
}),
|
||||
)
|
||||
.contract()
|
||||
|
||||
export const settingsGet = procedure("knowledge.settings.get")
|
||||
.kind("read")
|
||||
.scope("org")
|
||||
.permission("knowledge.settings.manage")
|
||||
.input(z.object({ ...inOrg }))
|
||||
.output(
|
||||
z.object({
|
||||
enabled: z.boolean(),
|
||||
backend: z.enum(KNOWLEDGE_BACKENDS).nullable(),
|
||||
}),
|
||||
)
|
||||
.contract()
|
||||
|
||||
/** Its checks and the managed link call the platform's APIs, so it is an effect, like the search. */
|
||||
export const settingsEnableDocumentAi = procedure("knowledge.settings.enableDocumentAi")
|
||||
.kind("effect")
|
||||
.scope("org")
|
||||
.permission("knowledge.settings.manage")
|
||||
.input(z.object({ ...inOrg }))
|
||||
.output(z.object({ enabled: z.boolean(), backend: z.enum(KNOWLEDGE_BACKENDS) }))
|
||||
.audit(DOCUMENT_AI_ENABLED)
|
||||
.contract()
|
||||
|
||||
export const settingsDisableDocumentAi = procedure("knowledge.settings.disableDocumentAi")
|
||||
.kind("write")
|
||||
.scope("org")
|
||||
.permission("knowledge.settings.manage")
|
||||
.input(z.object({ ...inOrg }))
|
||||
.output(z.object({ enabled: z.boolean() }))
|
||||
.audit(DOCUMENT_AI_DISABLED)
|
||||
.contract()
|
||||
|
||||
/** Whether this project indexes its uploaded PDFs automatically, and into which collection. */
|
||||
export const autoIndexGet = procedure("knowledge.autoindex.get")
|
||||
.kind("read")
|
||||
.scope("project")
|
||||
.permission("knowledge.knowledge.search")
|
||||
.input(z.object({ ...inProject }))
|
||||
.output(z.object({ collectionName: collectionName.nullable() }))
|
||||
.contract()
|
||||
|
||||
/**
|
||||
* Turns automatic PDF indexing on (into an existing collection) or off (`null`). The administrator who
|
||||
* turns it on is the one every indexing job re-checks: their permission is the job's authority.
|
||||
*/
|
||||
export const autoIndexSet = procedure("knowledge.autoindex.set")
|
||||
.kind("write")
|
||||
.scope("project")
|
||||
.permission("knowledge.knowledge.manage")
|
||||
.input(z.object({ ...inProject, collectionName: collectionName.nullable() }))
|
||||
.output(z.object({ collectionName: collectionName.nullable() }))
|
||||
.audit(AUTO_INDEX_CHANGED)
|
||||
.contract()
|
||||
|
||||
export const KNOWLEDGE_PROCEDURES = [
|
||||
collectionList,
|
||||
collectionCreate,
|
||||
documentIndex,
|
||||
documentList,
|
||||
documentGetMany,
|
||||
documentRemove,
|
||||
knowledgeSearch,
|
||||
knowledgeStatus,
|
||||
settingsGet,
|
||||
settingsEnableDocumentAi,
|
||||
settingsDisableDocumentAi,
|
||||
autoIndexGet,
|
||||
autoIndexSet,
|
||||
]
|
||||
|
||||
export type { DocumentStatus, ServiceState }
|
||||
@@ -0,0 +1,53 @@
|
||||
// The statuses a knowledge document moves through. `unavailable` and `disabled` are not rows: they
|
||||
// are the module's and the service's answers, and the UI maps all eight states from what a page
|
||||
// reads (the module flag, the service readiness, the document row).
|
||||
|
||||
export const DOCUMENT_STATUSES = ["queued", "indexing", "indexed", "failed", "cancelled"] as const
|
||||
export type DocumentStatus = (typeof DOCUMENT_STATUSES)[number]
|
||||
|
||||
/** The search backend a deployment binds: the platform's managed service, and no other (ADR-061). */
|
||||
export const KNOWLEDGE_BACKENDS = ["platform"] as const
|
||||
export type KnowledgeBackend = (typeof KNOWLEDGE_BACKENDS)[number]
|
||||
|
||||
/** The readiness states the search port answers (the managed toolset's readiness + unavailable). */
|
||||
export const SERVICE_STATES = ["registered", "indexing", "ready", "degraded", "disabled", "unavailable"] as const
|
||||
export type ServiceState = (typeof SERVICE_STATES)[number]
|
||||
|
||||
/** The enablement flow's typed refusal reasons, surfaced in the UI as states. */
|
||||
export const ENABLEMENT_REASONS = ["plan_required", "feature_disabled", "quota_exhausted"] as const
|
||||
export type EnablementReason = (typeof ENABLEMENT_REASONS)[number]
|
||||
|
||||
/**
|
||||
* Why the platform's knowledge service will not serve this deployment, as the pages show it. The service's
|
||||
* reason codes reach the module as these (adapters/refusals.ts): `scope_missing` is the service refusing
|
||||
* the deployment's token, whose `conusai.json` asks for `knowledge.collections:use`.
|
||||
*/
|
||||
export const KNOWLEDGE_REFUSALS = [
|
||||
"plan_required",
|
||||
"feature_disabled",
|
||||
"quota_exhausted",
|
||||
"budget_exhausted",
|
||||
"scope_missing",
|
||||
"indexing",
|
||||
"approval_required",
|
||||
"runtime_unavailable",
|
||||
"configuration_missing",
|
||||
] as const
|
||||
export type KnowledgeRefusal = (typeof KNOWLEDGE_REFUSALS)[number]
|
||||
|
||||
/** The indexing failure codes a document row records: the file's own, and a refusal waiting cannot fix. */
|
||||
export const INDEX_FAILURES = [
|
||||
"processing_exhausted",
|
||||
"configuration_missing",
|
||||
"not_indexable",
|
||||
"too_large",
|
||||
"not_text",
|
||||
"file_changed",
|
||||
"plan_required",
|
||||
"feature_disabled",
|
||||
"quota_exhausted",
|
||||
"budget_exhausted",
|
||||
"scope_missing",
|
||||
"approval_required",
|
||||
] as const
|
||||
export type IndexFailure = (typeof INDEX_FAILURES)[number]
|
||||
@@ -0,0 +1,72 @@
|
||||
// The automatic PDF intake (template lesson core-11): what the worker runs for one uploaded file the
|
||||
// files module handed to this module's processor. A project that opted in (`knowledge_auto_index`)
|
||||
// gets the PDF indexed into its collection as a document like any other, requested by the
|
||||
// administrator who opted in: the job runs for nobody else, so that administrator's admission is what
|
||||
// each step leaving the process checks again (`beforeIndex`). Anything else reads nothing.
|
||||
|
||||
import { AppError } from "@conusai-kit/core"
|
||||
import { getFile } from "@conusai-kit/files"
|
||||
import { AUTO_INDEX_CONTENT_TYPES, autoIndex, type DocumentView, documents } from "../contract/index.js"
|
||||
import { indexPassages } from "./index-job.js"
|
||||
import { type KnowledgeDeps, type KnowledgeScope, knowledgeEnabled } from "./ports.js"
|
||||
|
||||
/** What the files module's processor job carries about the upload. */
|
||||
export interface AutoIndexUpload {
|
||||
fileId: string
|
||||
contentType: string
|
||||
sha256: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Indexes one upload of an opted-in project, or answers null and reads nothing: a type it does not read,
|
||||
* a project that is not opted in or has knowledge off, a file that is gone, or one whose digest is not
|
||||
* the upload's. A redelivered job finds the document row and resumes it, so the work is done once.
|
||||
*/
|
||||
export async function autoIndexUpload(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
upload: AutoIndexUpload,
|
||||
options: { signal?: AbortSignal; finalAttempt?: boolean } = {},
|
||||
): Promise<DocumentView | null> {
|
||||
if (!(AUTO_INDEX_CONTENT_TYPES as readonly string[]).includes(upload.contentType)) return null
|
||||
const optIn = (await deps.data.forScope(scope).of(autoIndex).list({ limit: 1 })).items[0]
|
||||
if (!optIn) return null
|
||||
if (!(await knowledgeEnabled(deps, scope))) return null
|
||||
const file = await storedFile(deps, scope, upload.fileId)
|
||||
if (!file || file.sha256 !== upload.sha256) return null
|
||||
|
||||
const row = await deps.data.unitOfWork.run(scope, async (tx) => {
|
||||
const existing = (
|
||||
await tx.of(documents).list({
|
||||
where: [
|
||||
{ field: "collectionName", op: "eq", value: optIn.collectionName },
|
||||
{ field: "fileId", op: "eq", value: upload.fileId },
|
||||
],
|
||||
limit: 1,
|
||||
})
|
||||
).items[0]
|
||||
return (
|
||||
existing ??
|
||||
(await tx.of(documents).insert({
|
||||
collectionName: optIn.collectionName,
|
||||
fileId: upload.fileId,
|
||||
filename: file.filename,
|
||||
status: "queued",
|
||||
error: null,
|
||||
indexedAt: null,
|
||||
requestedBy: optIn.enabledBy,
|
||||
}))
|
||||
)
|
||||
})
|
||||
return indexPassages(deps, scope, row.id, options)
|
||||
}
|
||||
|
||||
/** The upload's file, or undefined once it is no longer stored (deleted, or on its way out). */
|
||||
async function storedFile(deps: KnowledgeDeps, scope: KnowledgeScope, fileId: string) {
|
||||
try {
|
||||
return await getFile(deps.files, scope, fileId)
|
||||
} catch (error) {
|
||||
if (error instanceof AppError && error.status === 404) return undefined
|
||||
throw error
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
// A project's automatic PDF indexing (template lesson core-11): an administrator opts the project in,
|
||||
// naming the collection its uploaded PDFs go into; no row is off. The administrator is recorded,
|
||||
// because the indexing job runs for nobody and re-checks that person's permission before each file.
|
||||
|
||||
import { forbidden } from "@conusai-kit/core"
|
||||
import { autoIndex } from "../contract/index.js"
|
||||
import { requireCollection } from "./collections.js"
|
||||
import { type KnowledgeDeps, type KnowledgeScope, knowledgeEnabled } from "./ports.js"
|
||||
|
||||
export interface AutoIndexView {
|
||||
/** The collection uploaded PDFs are indexed into, or null when automatic indexing is off. */
|
||||
collectionName: string | null
|
||||
}
|
||||
|
||||
export async function getAutoIndex(deps: Pick<KnowledgeDeps, "data">, scope: KnowledgeScope): Promise<AutoIndexView> {
|
||||
const row = (await deps.data.forScope(scope).of(autoIndex).list({ limit: 1 })).items[0]
|
||||
return { collectionName: row?.collectionName ?? null }
|
||||
}
|
||||
|
||||
/** Turns automatic indexing on into `collectionName` (which must exist), or off with `null`. */
|
||||
export async function setAutoIndex(
|
||||
deps: KnowledgeDeps & { now?: () => number },
|
||||
scope: KnowledgeScope,
|
||||
collectionName: string | null,
|
||||
): Promise<AutoIndexView> {
|
||||
if (collectionName !== null) {
|
||||
// The tenant-wall read first: another project's collection answers NotFound, never a policy refusal.
|
||||
await requireCollection(deps, scope, collectionName)
|
||||
if (!(await knowledgeEnabled(deps, scope))) throw forbidden("knowledge is switched off for this project")
|
||||
}
|
||||
await deps.data.unitOfWork.run(scope, async (tx) => {
|
||||
const row = (await tx.of(autoIndex).list({ limit: 1 })).items[0]
|
||||
if (collectionName === null) {
|
||||
if (row) await tx.of(autoIndex).delete(row.id)
|
||||
return
|
||||
}
|
||||
const patch = {
|
||||
collectionName,
|
||||
enabledBy: scope.actor.userId,
|
||||
enabledAt: new Date((deps.now ?? Date.now)()).toISOString(),
|
||||
}
|
||||
if (row) await tx.of(autoIndex).update(row.id, patch, row.version)
|
||||
else await tx.of(autoIndex).insert(patch)
|
||||
})
|
||||
return { collectionName }
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
// The collections: list them with their document counts, create one. A collection is a named
|
||||
// bucket; its name is the slug every procedure takes, so a page never handles an id.
|
||||
|
||||
import { conflict, notFound, validationFailed } from "@conusai-kit/core"
|
||||
import { type CollectionView, collections, documents } from "../contract/index.js"
|
||||
import type { KnowledgeDeps, KnowledgeScope } from "./ports.js"
|
||||
|
||||
const PAGE = 200
|
||||
|
||||
export async function listCollections(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
): Promise<{ collections: CollectionView[] }> {
|
||||
const found = []
|
||||
let cursor: string | undefined
|
||||
for (;;) {
|
||||
const page = await deps.data
|
||||
.forScope(scope)
|
||||
.of(collections)
|
||||
.list({
|
||||
order: [{ field: "createdAt", direction: "asc" }],
|
||||
limit: PAGE,
|
||||
...(cursor === undefined ? {} : { cursor }),
|
||||
})
|
||||
found.push(...page.items)
|
||||
if (!page.hasMore) break
|
||||
cursor = page.cursor
|
||||
}
|
||||
const views: CollectionView[] = []
|
||||
for (const row of found) {
|
||||
const counts = await documentCounts(deps, scope, row.name)
|
||||
views.push({
|
||||
name: row.name,
|
||||
documentCount: counts.total,
|
||||
indexedCount: counts.indexed,
|
||||
createdBy: row.createdBy,
|
||||
})
|
||||
}
|
||||
return { collections: views }
|
||||
}
|
||||
|
||||
async function documentCounts(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
collectionName: string,
|
||||
): Promise<{ total: number; indexed: number }> {
|
||||
let total = 0
|
||||
let indexed = 0
|
||||
let cursor: string | undefined
|
||||
for (;;) {
|
||||
const page = await deps.data
|
||||
.forScope(scope)
|
||||
.of(documents)
|
||||
.list({
|
||||
where: [{ field: "collectionName", op: "eq", value: collectionName }],
|
||||
limit: PAGE,
|
||||
...(cursor === undefined ? {} : { cursor }),
|
||||
})
|
||||
for (const row of page.items) {
|
||||
total += 1
|
||||
if (row.status === "indexed") indexed += 1
|
||||
}
|
||||
if (!page.hasMore) break
|
||||
cursor = page.cursor
|
||||
}
|
||||
return { total, indexed }
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a collection, idempotently: a name that already exists answers with the existing
|
||||
* collection and writes nothing, so a retried request never audits twice.
|
||||
*/
|
||||
export async function createCollection(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
name: string,
|
||||
): Promise<CollectionView> {
|
||||
const repos = deps.data.forScope(scope)
|
||||
const existing = (await repos.of(collections).list({ where: [{ field: "name", op: "eq", value: name }], limit: 1 }))
|
||||
.items[0]
|
||||
if (existing) {
|
||||
const counts = await documentCounts(deps, scope, name)
|
||||
return {
|
||||
name: existing.name,
|
||||
documentCount: counts.total,
|
||||
indexedCount: counts.indexed,
|
||||
createdBy: existing.createdBy,
|
||||
}
|
||||
}
|
||||
try {
|
||||
const row = await deps.data.unitOfWork.run(scope, async (tx) => {
|
||||
const inserted = await tx.of(collections).insert({ name, createdBy: scope.actor.userId })
|
||||
return inserted
|
||||
})
|
||||
return { name: row.name, documentCount: 0, indexedCount: 0, createdBy: row.createdBy }
|
||||
} catch (error) {
|
||||
// A concurrent create with the same name lost the race: answer the winner's row.
|
||||
if ((error as { status?: number }).status === 409) {
|
||||
const winner = (await repos.of(collections).list({ where: [{ field: "name", op: "eq", value: name }], limit: 1 }))
|
||||
.items[0]
|
||||
if (winner) {
|
||||
const counts = await documentCounts(deps, scope, name)
|
||||
return {
|
||||
name: winner.name,
|
||||
documentCount: counts.total,
|
||||
indexedCount: counts.indexed,
|
||||
createdBy: winner.createdBy,
|
||||
}
|
||||
}
|
||||
}
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/** One collection by name, or NotFound: the resolver's own read is the membership check. */
|
||||
export async function requireCollection(deps: KnowledgeDeps, scope: KnowledgeScope, name: string) {
|
||||
const row = (
|
||||
await deps.data
|
||||
.forScope(scope)
|
||||
.of(collections)
|
||||
.list({ where: [{ field: "name", op: "eq", value: name }], limit: 1 })
|
||||
).items[0]
|
||||
if (!row) throw notFound("no such collection")
|
||||
return row
|
||||
}
|
||||
|
||||
export const unknownCollection = (name: string): never => {
|
||||
throw validationFailed([{ path: "collectionName", message: "no such collection" }], `no collection ${name}`)
|
||||
}
|
||||
export const takenCollection = (): never => {
|
||||
throw conflict("this collection name is taken")
|
||||
}
|
||||
@@ -0,0 +1,255 @@
|
||||
// The documents: request an index (a web write that records the row and publishes the outbox
|
||||
// event the worker's subscriber turns into the index job), request a removal the same way, and
|
||||
// list a collection's documents with their progress states.
|
||||
|
||||
import { AppError, conflict, forbidden, notFound, validationFailed } from "@conusai-kit/core"
|
||||
import { getFile, getFileStorage } from "@conusai-kit/files"
|
||||
import {
|
||||
type DocumentView,
|
||||
documentIndexRequested,
|
||||
documentRemoveRequested,
|
||||
documents,
|
||||
INDEXABLE_CONTENT_TYPES,
|
||||
MAX_INDEX_BYTES,
|
||||
} from "../contract/index.js"
|
||||
import { requireCollection } from "./collections.js"
|
||||
import { type KnowledgeDeps, type KnowledgeScope, knowledgeEnabled } from "./ports.js"
|
||||
|
||||
const PAGE = 200
|
||||
|
||||
export const documentView = (row: {
|
||||
id: string
|
||||
collectionName: string
|
||||
fileId: string
|
||||
filename: string
|
||||
status: DocumentView["status"]
|
||||
error: string | null
|
||||
indexedAt: string | null
|
||||
requestedBy: string
|
||||
}): DocumentView => ({
|
||||
documentId: row.id,
|
||||
collectionName: row.collectionName,
|
||||
fileId: row.fileId,
|
||||
filename: row.filename,
|
||||
status: row.status,
|
||||
error: row.error,
|
||||
indexedAt: row.indexedAt,
|
||||
requestedBy: row.requestedBy,
|
||||
})
|
||||
|
||||
export async function listDocuments(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
collectionName: string,
|
||||
): Promise<{ documents: DocumentView[] }> {
|
||||
const found = []
|
||||
let cursor: string | undefined
|
||||
for (;;) {
|
||||
const page = await deps.data
|
||||
.forScope(scope)
|
||||
.of(documents)
|
||||
.list({
|
||||
where: [{ field: "collectionName", op: "eq", value: collectionName }],
|
||||
order: [{ field: "createdAt", direction: "desc" }],
|
||||
limit: PAGE,
|
||||
...(cursor === undefined ? {} : { cursor }),
|
||||
})
|
||||
found.push(...page.items)
|
||||
if (!page.hasMore) break
|
||||
cursor = page.cursor
|
||||
}
|
||||
return { documents: found.map((row) => documentView(row)) }
|
||||
}
|
||||
|
||||
/** The most ids one batch-get reads (contract C-05). */
|
||||
export const MAX_BATCH = 100
|
||||
|
||||
/**
|
||||
* The live collection's read-back: the documents with these ids, in whichever collection of the project.
|
||||
* A row this caller cannot see is not in the answer: the repository's tenant wall leaves it out.
|
||||
*/
|
||||
export async function getManyDocuments(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
documentIds: readonly string[],
|
||||
): Promise<DocumentView[]> {
|
||||
const page = await deps.data
|
||||
.forScope(scope)
|
||||
.of(documents)
|
||||
.list({ where: [{ field: "id", op: "in", values: [...new Set(documentIds)] }], limit: MAX_BATCH })
|
||||
return page.items.map((row) => documentView(row))
|
||||
}
|
||||
|
||||
/**
|
||||
* Asks for a document's indexing. The write is three things in one transaction: the durable row
|
||||
* (the UI's progress states and the job's idempotency key), the outbox event the worker's
|
||||
* subscriber enqueues the index job from, and nothing else — the call to the platform's knowledge
|
||||
* service happens in the worker, never in a request (AGENTS.md: long work is a job).
|
||||
*
|
||||
* Idempotent by (collection, file): asking again for a pair that is queued or indexing answers
|
||||
* the same row and publishes nothing; asking for an indexed or failed pair re-queues the one row,
|
||||
* so a retry can never duplicate a document.
|
||||
*/
|
||||
export async function requestIndex(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
input: { collectionName: string; fileId: string },
|
||||
): Promise<DocumentView> {
|
||||
// The tenant-wall reads come before the flag check: another tenant's ids answer NotFound, never
|
||||
// a policy refusal, so ids do not leak existence behind a disabled flag.
|
||||
await requireCollection(deps, scope, input.collectionName)
|
||||
const file = await getFile(deps.files, scope, input.fileId)
|
||||
if (!(INDEXABLE_CONTENT_TYPES as readonly string[]).includes(file.contentType)) {
|
||||
throw validationFailed(
|
||||
[{ path: "fileId", message: `${file.contentType} cannot be indexed; index a text file` }],
|
||||
"this file type is not indexable",
|
||||
)
|
||||
}
|
||||
if (file.size > MAX_INDEX_BYTES) {
|
||||
throw validationFailed(
|
||||
[{ path: "fileId", message: `the file is larger than ${MAX_INDEX_BYTES} bytes` }],
|
||||
"the file is too large to index",
|
||||
)
|
||||
}
|
||||
if (!(await knowledgeEnabled(deps, scope))) throw forbidden("knowledge is switched off for this project")
|
||||
|
||||
return deps.data.unitOfWork.run(scope, async (tx) => {
|
||||
const existing = (
|
||||
await tx.of(documents).list({
|
||||
where: [
|
||||
{ field: "collectionName", op: "eq", value: input.collectionName },
|
||||
{ field: "fileId", op: "eq", value: input.fileId },
|
||||
],
|
||||
limit: 1,
|
||||
})
|
||||
).items[0]
|
||||
if (existing && (existing.status === "queued" || existing.status === "indexing")) {
|
||||
return documentView(existing)
|
||||
}
|
||||
const row =
|
||||
existing !== undefined
|
||||
? await tx
|
||||
.of(documents)
|
||||
.update(
|
||||
existing.id,
|
||||
{ status: "queued", error: null, indexedAt: null, requestedBy: scope.actor.userId },
|
||||
existing.version,
|
||||
)
|
||||
: await tx.of(documents).insert({
|
||||
collectionName: input.collectionName,
|
||||
fileId: input.fileId,
|
||||
filename: file.filename,
|
||||
status: "queued",
|
||||
error: null,
|
||||
indexedAt: null,
|
||||
requestedBy: scope.actor.userId,
|
||||
})
|
||||
await tx.publish(documentIndexRequested, {
|
||||
orgId: scope.orgId,
|
||||
workspaceId: scope.workspaceId,
|
||||
projectId: scope.projectId,
|
||||
documentId: row.id,
|
||||
collectionName: input.collectionName,
|
||||
fileId: input.fileId,
|
||||
})
|
||||
return documentView(row)
|
||||
})
|
||||
}
|
||||
|
||||
/** Asks for a document's removal: the row and the event in one transaction, the job in the worker. */
|
||||
export async function requestRemove(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
input: { collectionName: string; fileId: string; storageKey?: string },
|
||||
): Promise<DocumentView> {
|
||||
const current = (
|
||||
await deps.data
|
||||
.forScope(scope)
|
||||
.of(documents)
|
||||
.list({
|
||||
where: [
|
||||
{ field: "collectionName", op: "eq", value: input.collectionName },
|
||||
{ field: "fileId", op: "eq", value: input.fileId },
|
||||
],
|
||||
limit: 1,
|
||||
})
|
||||
).items[0]
|
||||
// The tenant-wall read comes first, as in requestIndex: another tenant's ids answer NotFound.
|
||||
if (!current) throw notFound("no such document")
|
||||
if (current.status === "cancelled") return documentView(current)
|
||||
// Every removal names the object it removes, resolved when it is asked for, not when the job runs: by then the
|
||||
// file may be gone (F407). A deletion brings the key it had; a person's removal reads it from the stored file.
|
||||
const storageKey = input.storageKey ?? (await storageKeyOf(deps, scope, input.fileId))
|
||||
return deps.data.unitOfWork.run(scope, async (tx) => {
|
||||
const row = (
|
||||
await tx.of(documents).list({
|
||||
where: [
|
||||
{ field: "collectionName", op: "eq", value: input.collectionName },
|
||||
{ field: "fileId", op: "eq", value: input.fileId },
|
||||
],
|
||||
limit: 1,
|
||||
})
|
||||
).items[0]
|
||||
if (!row) throw notFound("no such document")
|
||||
if (row.status === "cancelled") return documentView(row)
|
||||
const next = await tx.of(documents).update(row.id, { status: "cancelled", error: null }, row.version)
|
||||
await tx.publish(documentRemoveRequested, {
|
||||
orgId: scope.orgId,
|
||||
workspaceId: scope.workspaceId,
|
||||
projectId: scope.projectId,
|
||||
documentId: row.id,
|
||||
collectionName: input.collectionName,
|
||||
fileId: input.fileId,
|
||||
storageKey,
|
||||
})
|
||||
return documentView(next)
|
||||
})
|
||||
}
|
||||
|
||||
/** The document row a job resumes by; NotFound when another tenant's id was named. */
|
||||
export async function documentById(deps: KnowledgeDeps, scope: KnowledgeScope, documentId: string) {
|
||||
const row = await deps.data.forScope(scope).of(documents).get(documentId)
|
||||
if (!row) throw notFound("no such document")
|
||||
return row
|
||||
}
|
||||
|
||||
/**
|
||||
* The stored file's object key. A file on its way out (being deleted, or gone) is refused rather than removed
|
||||
* unkeyed: its deletion publishes `FileDeleted` with the key, and that removal finds the document still active.
|
||||
*/
|
||||
async function storageKeyOf(deps: KnowledgeDeps, scope: KnowledgeScope, fileId: string): Promise<string> {
|
||||
try {
|
||||
return (await getFileStorage(deps.files, scope, fileId)).storageKey
|
||||
} catch (error) {
|
||||
if (error instanceof AppError && error.status === 404) {
|
||||
throw conflict("the file is being deleted; its indexed text is removed with it")
|
||||
}
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/** A source deletion revokes every collection's citation and durably requests cleanup.
|
||||
* Does not require processing enablement: deleting retained data is independent of inference.
|
||||
* `storageKey` is the key the deleted file had (`FileDeleted`): its row is gone, so the removal cannot look it up. */
|
||||
export async function requestSourceRemoval(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
fileId: string,
|
||||
storageKey: string,
|
||||
): Promise<void> {
|
||||
let cursor: string | undefined
|
||||
for (;;) {
|
||||
const page = await deps.data
|
||||
.forScope(scope)
|
||||
.of(documents)
|
||||
.list({
|
||||
where: [{ field: "fileId", op: "eq", value: fileId }],
|
||||
limit: PAGE,
|
||||
...(cursor === undefined ? {} : { cursor }),
|
||||
})
|
||||
for (const row of page.items)
|
||||
await requestRemove(deps, scope, { collectionName: row.collectionName, fileId, storageKey })
|
||||
if (!page.hasMore) break
|
||||
cursor = page.cursor
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,280 @@
|
||||
// The index job's body: what the worker runs for one document. Read the file's bytes, hand them to
|
||||
// the search service, and record the outcome on the durable row.
|
||||
// Transient failures (the service hiccups) throw, and the queue retries the job; the row the
|
||||
// retry resumes from makes the retry safe. Permanent failures are recorded on the row with a
|
||||
// stable code, so a person sees why a document failed and a retried request does not pay twice.
|
||||
// The automatic PDF intake (core-11) runs the same lifecycle with another send: the PDF is read into
|
||||
// passages and each passage is indexed on its own (`passageSend`).
|
||||
|
||||
import { internal } from "@conusai-kit/core"
|
||||
import { downloadFile, getFile, readAll, sha256Hex } from "@conusai-kit/files"
|
||||
import {
|
||||
type DocumentView,
|
||||
documents,
|
||||
INDEX_FAILURES,
|
||||
type IndexFailure,
|
||||
MAX_AUTO_INDEX_BYTES,
|
||||
MAX_INDEX_BYTES,
|
||||
passageSets,
|
||||
} from "../contract/index.js"
|
||||
import { documentById, documentView } from "./documents.js"
|
||||
import { type KnowledgeDeps, type KnowledgePassage, type KnowledgeScope, knowledgeEnabled } from "./ports.js"
|
||||
|
||||
type DocumentRow = Awaited<ReturnType<typeof documentById>>
|
||||
|
||||
/** What one run sends between the row's "indexing" and its outcome; it throws to fail or to retry. */
|
||||
type Send = (row: DocumentRow, signal: AbortSignal | undefined) => Promise<void>
|
||||
|
||||
export async function indexDocument(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
documentId: string,
|
||||
options: { signal?: AbortSignal; finalAttempt?: boolean } = {},
|
||||
): Promise<DocumentView> {
|
||||
return runIndex(deps, scope, documentId, () => textSend(deps, scope), options)
|
||||
}
|
||||
|
||||
/** The automatic PDF intake's run of one document: the PDF read into passages, each indexed on its own. */
|
||||
export async function indexPassages(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
documentId: string,
|
||||
options: { signal?: AbortSignal; finalAttempt?: boolean } = {},
|
||||
): Promise<DocumentView> {
|
||||
return runIndex(deps, scope, documentId, () => passageSend(deps, scope), options)
|
||||
}
|
||||
|
||||
/** The whole file as text, to the search service. */
|
||||
function textSend(deps: KnowledgeDeps, scope: KnowledgeScope): Send {
|
||||
const service = deps.search
|
||||
if (!service) throw internal("the knowledge search service is not bound in this process")
|
||||
return async (row, signal) => {
|
||||
const bytes = await readFileBytes(deps, scope, row.fileId, MAX_INDEX_BYTES)
|
||||
const text = decodeText(bytes)
|
||||
await deps.beforeIndex?.(row.requestedBy)
|
||||
signal?.throwIfAborted()
|
||||
await service.index(scope, {
|
||||
collectionName: row.collectionName,
|
||||
fileId: row.fileId,
|
||||
filename: row.filename,
|
||||
bytes: new TextEncoder().encode(text),
|
||||
...(signal ? { signal } : {}),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A PDF, read by the platform's document reader into passages, each indexed as one item. What is about to
|
||||
* be indexed is recorded before any of it is sent, so a run the service refuses partway can still be
|
||||
* removed passage by passage. The read is keyed by the file and its digest: a retry reads the same job.
|
||||
*/
|
||||
function passageSend(deps: KnowledgeDeps, scope: KnowledgeScope): Send {
|
||||
const { reader, passages, enablement } = deps
|
||||
if (!reader || !passages) throw internal("the document reader and the passage index are not bound in this process")
|
||||
if (!enablement) throw internal("the enablement platform is not bound in this process")
|
||||
return async (row, signal) => {
|
||||
const file = await getFile(deps.files, scope, row.fileId)
|
||||
if (file.size > MAX_AUTO_INDEX_BYTES)
|
||||
throw Object.assign(new Error("the file is too large to read"), { permanent: "too_large" })
|
||||
// The organization's eligibility and quota, as the enablement flow checks them, before the read is paid
|
||||
// for: a rollout kill switch or a spent quota stops the next PDF (root's core-11 decision). The service
|
||||
// still refuses past its quota at call time; this check only stops earlier.
|
||||
const eligibility = await enablement.checkEligibility(scope)
|
||||
if (!eligibility.ok)
|
||||
throw Object.assign(new Error(eligibility.detail ?? eligibility.reason), { permanent: eligibility.reason })
|
||||
if ((await enablement.indexingQuota(scope)) <= 0)
|
||||
throw Object.assign(new Error("the indexing volume for today is spent"), { permanent: "quota_exhausted" })
|
||||
signal?.throwIfAborted()
|
||||
const bytes = await readFileBytes(deps, scope, row.fileId, MAX_AUTO_INDEX_BYTES)
|
||||
await deps.beforeIndex?.(row.requestedBy)
|
||||
signal?.throwIfAborted()
|
||||
const read = await reader.read(bytes, {
|
||||
filename: row.filename,
|
||||
idempotencyKey: `knowledge.autoindex.${row.fileId}.${file.sha256}`,
|
||||
...(signal ? { signal } : {}),
|
||||
})
|
||||
const documentRef = await passages.reference(scope, row.fileId)
|
||||
await recordPassages(deps, scope, row, documentRef, read)
|
||||
await deps.beforeIndex?.(row.requestedBy)
|
||||
signal?.throwIfAborted()
|
||||
await passages.index(scope, {
|
||||
collectionName: row.collectionName,
|
||||
documentRef,
|
||||
fileId: row.fileId,
|
||||
passages: read,
|
||||
...(signal ? { signal } : {}),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/** The passages per page, comma-separated from page 1: what `passagesOf` turns back into references. */
|
||||
const pagesOf = (read: readonly KnowledgePassage[]): string => {
|
||||
const counts: number[] = []
|
||||
for (const passage of read) {
|
||||
while (counts.length < passage.page) counts.push(0)
|
||||
counts[passage.page - 1] = (counts[passage.page - 1] ?? 0) + 1
|
||||
}
|
||||
return counts.join(",")
|
||||
}
|
||||
|
||||
const passagesOf = (pages: string): Array<{ page: number; index: number }> =>
|
||||
pages === ""
|
||||
? []
|
||||
: pages
|
||||
.split(",")
|
||||
.flatMap((count, at) => Array.from({ length: Number(count) }, (_, index) => ({ page: at + 1, index })))
|
||||
|
||||
async function recordPassages(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
row: DocumentRow,
|
||||
documentRef: string,
|
||||
read: readonly KnowledgePassage[],
|
||||
): Promise<void> {
|
||||
const fields = { documentRef, pages: pagesOf(read) }
|
||||
await deps.data.unitOfWork.run(scope, async (tx) => {
|
||||
const existing = (await tx.of(passageSets).list(passageSetOf(row))).items[0]
|
||||
if (existing) await tx.of(passageSets).update(existing.id, fields, existing.version)
|
||||
else await tx.of(passageSets).insert({ collectionName: row.collectionName, fileId: row.fileId, ...fields })
|
||||
})
|
||||
}
|
||||
|
||||
/** The query of a document's passage record: one per (collection, file), as the document row. */
|
||||
const passageSetOf = (row: { collectionName: string; fileId: string }) => ({
|
||||
where: [
|
||||
{ field: "collectionName" as const, op: "eq" as const, value: row.collectionName },
|
||||
{ field: "fileId" as const, op: "eq" as const, value: row.fileId },
|
||||
],
|
||||
limit: 1,
|
||||
})
|
||||
|
||||
async function runIndex(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
documentId: string,
|
||||
sendOf: () => Send,
|
||||
options: { signal?: AbortSignal; finalAttempt?: boolean },
|
||||
): Promise<DocumentView> {
|
||||
const row = await documentById(deps, scope, documentId)
|
||||
if (row.status === "indexed" || row.status === "cancelled" || row.status === "failed") {
|
||||
// A redelivered job or a crashed retry: the row says where the work stands.
|
||||
return {
|
||||
documentId: row.id,
|
||||
collectionName: row.collectionName,
|
||||
fileId: row.fileId,
|
||||
filename: row.filename,
|
||||
status: row.status,
|
||||
error: row.error,
|
||||
indexedAt: row.indexedAt,
|
||||
requestedBy: row.requestedBy,
|
||||
}
|
||||
}
|
||||
const send = sendOf()
|
||||
const indexing = await deps.data
|
||||
.forScope(scope)
|
||||
.of(documents)
|
||||
.update(row.id, { status: "indexing", error: null }, row.version)
|
||||
|
||||
try {
|
||||
options.signal?.throwIfAborted()
|
||||
if (!(await knowledgeEnabled(deps, scope)))
|
||||
throw Object.assign(new Error("processing disabled"), { permanent: "feature_disabled" })
|
||||
await deps.beforeIndex?.(row.requestedBy)
|
||||
options.signal?.throwIfAborted()
|
||||
await send(indexing, options.signal)
|
||||
options.signal?.throwIfAborted()
|
||||
const current = await deps.data.forScope(scope).of(documents).get(indexing.id)
|
||||
if (!current || current.status === "cancelled") return documentView({ ...indexing, status: "cancelled" })
|
||||
const done = await deps.data
|
||||
.forScope(scope)
|
||||
.of(documents)
|
||||
.update(indexing.id, { status: "indexed", indexedAt: new Date().toISOString(), error: null }, indexing.version)
|
||||
return {
|
||||
documentId: done.id,
|
||||
collectionName: done.collectionName,
|
||||
fileId: done.fileId,
|
||||
filename: done.filename,
|
||||
status: done.status as DocumentView["status"],
|
||||
error: done.error,
|
||||
indexedAt: done.indexedAt,
|
||||
requestedBy: done.requestedBy,
|
||||
}
|
||||
} catch (error) {
|
||||
if (options.signal?.aborted) throw error
|
||||
const permanent = permanentFailure(error) ?? (options.finalAttempt === true ? "processing_exhausted" : null)
|
||||
if (permanent === null) throw error
|
||||
const current = await deps.data.forScope(scope).of(documents).get(indexing.id)
|
||||
if (!current || current.status === "cancelled") return documentView({ ...indexing, status: "cancelled" })
|
||||
const failed = await deps.data
|
||||
.forScope(scope)
|
||||
.of(documents)
|
||||
.update(indexing.id, { status: "failed", error: permanent }, indexing.version)
|
||||
return {
|
||||
documentId: failed.id,
|
||||
collectionName: failed.collectionName,
|
||||
fileId: failed.fileId,
|
||||
filename: failed.filename,
|
||||
status: "failed",
|
||||
error: failed.error,
|
||||
indexedAt: failed.indexedAt,
|
||||
requestedBy: failed.requestedBy,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export async function removeDocument(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
documentId: string,
|
||||
storageKey?: string,
|
||||
): Promise<void> {
|
||||
const row = await deps.data.forScope(scope).of(documents).get(documentId)
|
||||
if (!row) return
|
||||
const set = (await deps.data.forScope(scope).of(passageSets).list(passageSetOf(row))).items[0]
|
||||
if (set) {
|
||||
if (!deps.passages) throw internal("the passage index is not bound in this process")
|
||||
await deps.passages.remove(scope, { documentRef: set.documentRef, passages: passagesOf(set.pages) })
|
||||
await deps.data.unitOfWork.run(scope, async (tx) => {
|
||||
await tx.of(documents).delete(row.id)
|
||||
await tx.of(passageSets).delete(set.id)
|
||||
})
|
||||
return
|
||||
}
|
||||
const service = deps.search
|
||||
if (!service) throw internal("the knowledge search service is not bound in this process")
|
||||
await service.remove(scope, {
|
||||
collectionName: row.collectionName,
|
||||
fileId: row.fileId,
|
||||
...(storageKey === undefined ? {} : { storageKey }),
|
||||
})
|
||||
await deps.data.forScope(scope).of(documents).delete(row.id)
|
||||
}
|
||||
|
||||
async function readFileBytes(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
fileId: string,
|
||||
maxBytes: number,
|
||||
): Promise<Uint8Array> {
|
||||
const file = await getFile(deps.files, scope, fileId)
|
||||
const response = await downloadFile(deps.files, scope, fileId)
|
||||
if (!response.body) throw Object.assign(new Error("source missing"), { permanent: "not_indexable" })
|
||||
const bytes = await readAll(response.body, maxBytes)
|
||||
if (!file.sha256 || (await sha256Hex(bytes)) !== file.sha256)
|
||||
throw Object.assign(new Error("source changed"), { permanent: "file_changed" })
|
||||
return bytes
|
||||
}
|
||||
|
||||
function decodeText(bytes: Uint8Array): string {
|
||||
try {
|
||||
return new TextDecoder("utf-8", { fatal: true }).decode(bytes)
|
||||
} catch {
|
||||
throw Object.assign(new Error("the file is not valid UTF-8 text"), { permanent: "not_text" })
|
||||
}
|
||||
}
|
||||
|
||||
/** Maps an indexing failure to the stable code recorded on the row; null when it is transient. */
|
||||
function permanentFailure(error: unknown): string | null {
|
||||
const code = (error as { permanent?: unknown }).permanent
|
||||
return (INDEX_FAILURES as readonly unknown[]).includes(code) ? (code as IndexFailure) : null
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
// The ports the knowledge module's use cases are written against, and the tokens the composition
|
||||
// roots bind. Core imports no SDK and no adapter (spec §2): the search service and the enablement
|
||||
// platform are shapes, and adapters/ is the only place an SDK exists.
|
||||
|
||||
import { type Container, type TenantScope, token } from "@conusai-kit/core"
|
||||
import type { DataSource } from "@conusai-kit/data"
|
||||
import type { FilesDeps } from "@conusai-kit/files"
|
||||
import type { EnablementReason, KnowledgeRefusal, ServiceState } from "../contract/index.js"
|
||||
import { config, KNOWLEDGE_FLAG_DEFAULTS } from "../contract/index.js"
|
||||
|
||||
/** A project scope: collections and documents live at this level. */
|
||||
export type KnowledgeScope = TenantScope & { workspaceId: string; projectId: string }
|
||||
|
||||
/** An org scope: the enablement configuration is the organization's own. */
|
||||
export type KnowledgeOrgScope = TenantScope & { orgId: string }
|
||||
|
||||
export interface KnowledgeDeps {
|
||||
readonly data: DataSource
|
||||
readonly files: FilesDeps
|
||||
readonly flags: KnowledgeFlags
|
||||
readonly beforeIndex?: (subjectUserId: string) => Promise<void>
|
||||
/**
|
||||
* The search service. Present for the effect procedure (search); a read or write procedure's deps
|
||||
* never carry it, because resolving an effect port from either is a kind violation — the write
|
||||
* only records the row and publishes the event, and the status read asks `KNOWLEDGE_READINESS`.
|
||||
*/
|
||||
readonly search?: KnowledgeSearchService | undefined
|
||||
/** The document reader and the passage index, for the automatic PDF intake (core-11); bound in the worker. */
|
||||
readonly reader?: KnowledgeReader | undefined
|
||||
readonly passages?: KnowledgePassageIndex | undefined
|
||||
/** The organization's eligibility and indexing quota, which the PDF intake checks before each read. */
|
||||
readonly enablement?: DocumentAiPlatform | undefined
|
||||
}
|
||||
|
||||
/** One cited search answer. */
|
||||
export interface KnowledgeHit {
|
||||
fileId: string
|
||||
filename: string
|
||||
collectionName: string
|
||||
/** 0..1; higher is closer. */
|
||||
score: number
|
||||
chunk: number
|
||||
snippet: string
|
||||
page?: number
|
||||
blockId?: string
|
||||
chunkId?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* The search service: index a document, search a collection with citations, remove a document, and
|
||||
* report readiness. Its one adapter is the platform's managed documents-index service (ADR-061), which
|
||||
* owns chunking, embeddings and the vector store. Every call leaves the process, so this port is an
|
||||
* effect. `refusal` is why the service will not serve this deployment, when that is the answer.
|
||||
*/
|
||||
export interface KnowledgeSearchService {
|
||||
readiness(scope: KnowledgeScope): Promise<{ state: ServiceState; reasons: string[]; refusal?: KnowledgeRefusal }>
|
||||
index(
|
||||
scope: KnowledgeScope,
|
||||
input: { collectionName: string; fileId: string; filename: string; bytes: Uint8Array; signal?: AbortSignal },
|
||||
): Promise<void>
|
||||
search(
|
||||
scope: KnowledgeScope,
|
||||
input: { collectionName: string; query: string; limit: number; actorId: string },
|
||||
): Promise<readonly KnowledgeHit[]>
|
||||
/** `storageKey` names the object, resolved when the removal was asked for; only an older record lacks it, and then the file is looked up. */
|
||||
remove(scope: KnowledgeScope, input: { collectionName: string; fileId: string; storageKey?: string }): Promise<void>
|
||||
}
|
||||
|
||||
/**
|
||||
* One passage of a read PDF, as the automatic intake indexes it: its page (1-based), its place on that
|
||||
* page (0-based; with the page, its stable name across reads of the same document), the document's own
|
||||
* block ids it holds lines of, and its text.
|
||||
*/
|
||||
export interface KnowledgePassage {
|
||||
readonly page: number
|
||||
readonly index: number
|
||||
readonly blockIds: readonly string[]
|
||||
readonly text: string
|
||||
}
|
||||
|
||||
/**
|
||||
* The platform's document reader, as the automatic intake uses it: a PDF in, its passages out. The
|
||||
* same `idempotencyKey` reads the same document once. A document it can never make passages of throws
|
||||
* with `permanent: "not_indexable"`; anything else is transient, and the queue retries the job.
|
||||
*/
|
||||
export interface KnowledgeReader {
|
||||
read(
|
||||
bytes: Uint8Array,
|
||||
options: { filename: string; idempotencyKey: string; signal?: AbortSignal },
|
||||
): Promise<readonly KnowledgePassage[]>
|
||||
}
|
||||
|
||||
/**
|
||||
* The knowledge service's inline index, one item per passage, each under the file's reference plus its
|
||||
* page and place. The reference is asked for first, so the caller can record what it is about to index
|
||||
* before any of it is sent, and `remove` takes it back because the file may be gone by then: the service
|
||||
* removes a reference only exactly, so removal names every passage that was indexed.
|
||||
*/
|
||||
export interface KnowledgePassageIndex {
|
||||
reference(scope: KnowledgeScope, fileId: string): Promise<string>
|
||||
index(
|
||||
scope: KnowledgeScope,
|
||||
input: {
|
||||
collectionName: string
|
||||
documentRef: string
|
||||
fileId: string
|
||||
passages: readonly KnowledgePassage[]
|
||||
signal?: AbortSignal
|
||||
},
|
||||
): Promise<void>
|
||||
remove(
|
||||
scope: KnowledgeScope,
|
||||
input: { documentRef: string; passages: ReadonlyArray<Pick<KnowledgePassage, "page" | "index">> },
|
||||
): Promise<void>
|
||||
}
|
||||
|
||||
/**
|
||||
* What the status read asks of the deployment's search backend. It is a port of its own because a read
|
||||
* handler may not resolve `KNOWLEDGE_SEARCH`, and must not build it: a factory runs with the resolving
|
||||
* handler's restrictions, so a search service built for a read would hold a read-only data source.
|
||||
*/
|
||||
export type KnowledgeReadiness = Pick<KnowledgeSearchService, "readiness">
|
||||
|
||||
/**
|
||||
* The enablement platform: the checks the enablement flow needs, over the platform's own APIs.
|
||||
* `checkEligibility` is where the plan capability and the rollout flags surface as typed reasons;
|
||||
* `indexingQuota` answers today's remaining indexing volume.
|
||||
*/
|
||||
export interface DocumentAiPlatform {
|
||||
checkEligibility(
|
||||
scope: KnowledgeOrgScope,
|
||||
): Promise<{ ok: true } | { ok: false; reason: EnablementReason; detail?: string }>
|
||||
indexingQuota(scope: KnowledgeOrgScope): Promise<number>
|
||||
}
|
||||
|
||||
/** What a flag service answers for the project a call runs in. */
|
||||
export interface KnowledgeFlags {
|
||||
enabled(flag: keyof typeof KNOWLEDGE_FLAG_DEFAULTS, scope: KnowledgeScope): boolean | Promise<boolean>
|
||||
}
|
||||
|
||||
/**
|
||||
* Bound by an app that evaluates flags (the platform's flag service). Unbound: knowledge is on for an
|
||||
* organization once an administrator has enabled document AI, and off until then (`enablementFlags`).
|
||||
*/
|
||||
export const KNOWLEDGE_FLAGS = token<KnowledgeFlags>("knowledge.flags")
|
||||
|
||||
/** The organization's own switch: the row `knowledge.settings.enableDocumentAi` writes. */
|
||||
export const enablementFlags = (data: DataSource): KnowledgeFlags => ({
|
||||
enabled: async (flag, scope) => {
|
||||
const row = (await data.forScope(scope).of(config).list({ limit: 1 })).items[0]
|
||||
return row?.enabled ?? KNOWLEDGE_FLAG_DEFAULTS[flag]
|
||||
},
|
||||
})
|
||||
|
||||
export const flagsOf = (container: Container, data: DataSource): KnowledgeFlags =>
|
||||
container.has(KNOWLEDGE_FLAGS) ? container.resolve(KNOWLEDGE_FLAGS) : enablementFlags(data)
|
||||
|
||||
/** Bound by the worker and the web shell with the module's search backend adapter. */
|
||||
export const KNOWLEDGE_SEARCH = token<KnowledgeSearchService>("knowledge.search", { access: "effect" })
|
||||
|
||||
/** Bound by the worker with the platform's document reader: the automatic PDF intake reads through it. */
|
||||
export const KNOWLEDGE_READER = token<KnowledgeReader>("knowledge.reader", { access: "effect" })
|
||||
|
||||
/** Bound by the worker with the knowledge service's inline index: the PDF intake's passages, and their removal. */
|
||||
export const KNOWLEDGE_PASSAGES = token<KnowledgePassageIndex>("knowledge.passages", { access: "effect" })
|
||||
|
||||
/** Bound by the web shell with the readiness of the service `KNOWLEDGE_SEARCH` serves. */
|
||||
export const KNOWLEDGE_READINESS = token<KnowledgeReadiness>("knowledge.readiness")
|
||||
|
||||
/**
|
||||
* Bound by the web shell (the enablement flow runs in a request) and by the worker (the PDF intake checks
|
||||
* eligibility and quota again before each read, core-11).
|
||||
*/
|
||||
export const KNOWLEDGE_ENABLEMENT = token<DocumentAiPlatform>("knowledge.enablement", { access: "effect" })
|
||||
|
||||
/** The per-run scope, under the module's own name (decision AT-F6). */
|
||||
export const KNOWLEDGE_SCOPE = token<TenantScope>("knowledge.scope")
|
||||
|
||||
export const knowledgeEnabled = async (deps: Pick<KnowledgeDeps, "flags">, scope: KnowledgeScope): Promise<boolean> =>
|
||||
deps.flags.enabled("knowledge.enabled", scope)
|
||||
@@ -0,0 +1,65 @@
|
||||
// The search: one collection, one query, cited answers. The query leaves the process (the procedure
|
||||
// is an effect), the ranking is the platform's service, and every answer carries where it came from:
|
||||
// the file, the chunk, and the snippet a person reads.
|
||||
|
||||
import { AppError, forbidden, internal } from "@conusai-kit/core"
|
||||
import { getFile } from "@conusai-kit/files"
|
||||
import { documents, MAX_SEARCH_RESULTS, type SearchResult } from "../contract/index.js"
|
||||
import { requireCollection } from "./collections.js"
|
||||
import { type KnowledgeDeps, type KnowledgeScope, knowledgeEnabled } from "./ports.js"
|
||||
|
||||
export async function searchKnowledge(
|
||||
deps: KnowledgeDeps,
|
||||
scope: KnowledgeScope,
|
||||
input: { collectionName: string; query: string; limit?: number | undefined },
|
||||
): Promise<{ results: SearchResult[] }> {
|
||||
if (!(await knowledgeEnabled(deps, scope))) throw forbidden("knowledge is switched off for this project")
|
||||
await requireCollection(deps, scope, input.collectionName)
|
||||
const service = deps.search
|
||||
if (!service) throw internal("the knowledge search service is not bound in this process")
|
||||
const limit = Math.min(input.limit ?? 10, MAX_SEARCH_RESULTS)
|
||||
const hits = await service.search(scope, {
|
||||
collectionName: input.collectionName,
|
||||
query: input.query,
|
||||
limit,
|
||||
actorId: scope.actor.userId,
|
||||
})
|
||||
const results: SearchResult[] = []
|
||||
for (const hit of hits) {
|
||||
// The current scoped source of truth revokes a citation before the remote deletion job finishes.
|
||||
const current = (
|
||||
await deps.data
|
||||
.forScope(scope)
|
||||
.of(documents)
|
||||
.list({
|
||||
where: [
|
||||
{ field: "collectionName", op: "eq", value: input.collectionName },
|
||||
{ field: "fileId", op: "eq", value: hit.fileId },
|
||||
{ field: "status", op: "eq", value: "indexed" },
|
||||
],
|
||||
limit: 1,
|
||||
})
|
||||
).items[0]
|
||||
if (!current || hit.collectionName !== input.collectionName) continue
|
||||
// The original file can be deleted before its outbox cleanup is delivered.
|
||||
// Current scoped file visibility is mandatory, independent of stale index rows.
|
||||
try {
|
||||
await getFile(deps.files, scope, hit.fileId)
|
||||
} catch (error) {
|
||||
if (error instanceof AppError && error.status === 404) continue
|
||||
throw error
|
||||
}
|
||||
results.push({
|
||||
fileId: hit.fileId,
|
||||
filename: hit.filename,
|
||||
collectionName: hit.collectionName,
|
||||
snippet: hit.snippet,
|
||||
score: hit.score,
|
||||
chunk: hit.chunk,
|
||||
...(hit.page === undefined ? {} : { page: hit.page }),
|
||||
...(hit.blockId === undefined ? {} : { blockId: hit.blockId }),
|
||||
...(hit.chunkId === undefined ? {} : { chunkId: hit.chunkId }),
|
||||
})
|
||||
}
|
||||
return { results }
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
// The enablement flow: one idempotent procedure pair an organization administrator runs, with the
|
||||
// platform's checks in front of any configuration change and every denial a typed reason code the
|
||||
// UI maps to an upgrade or policy state.
|
||||
//
|
||||
// enableDocumentAi: already enabled -> answers the current configuration, writes nothing.
|
||||
// Otherwise: eligibility (plan capability + rollout flags) -> quota -> one transaction persists
|
||||
// the configuration. Nothing is provisioned: retrieval is the platform's service (ADR-061), and an
|
||||
// app keeps no vector store. A failure at any step writes nothing and is retirable: the same call
|
||||
// again resumes from the first failing check.
|
||||
//
|
||||
// disableDocumentAi: preserves everything — collections, documents, the service's index and the
|
||||
// recorded configuration stay; only `enabled` flips, so re-enabling resumes without duplication.
|
||||
|
||||
import { AppError } from "@conusai-kit/core"
|
||||
import { config, type EnablementReason, type KnowledgeBackend } from "../contract/index.js"
|
||||
import type { DocumentAiPlatform, KnowledgeOrgScope } from "./ports.js"
|
||||
|
||||
/** A typed enablement denial: the problem's code is the stable reason the UI maps to a state. */
|
||||
export function enablementDenied(reason: EnablementReason, detail?: string): AppError {
|
||||
const status = reason === "quota_exhausted" ? 429 : 403
|
||||
const title =
|
||||
reason === "plan_required"
|
||||
? "The organization's plan does not include document AI"
|
||||
: reason === "feature_disabled"
|
||||
? "Document AI is disabled by policy"
|
||||
: "The indexing quota is exhausted"
|
||||
return new AppError({
|
||||
type: `/problems/${reason}`,
|
||||
title,
|
||||
status,
|
||||
code: reason,
|
||||
...(detail === undefined ? {} : { detail }),
|
||||
})
|
||||
}
|
||||
|
||||
export interface SettingsView {
|
||||
enabled: boolean
|
||||
/** Who serves this organization once it has a configuration: only the platform does. */
|
||||
backend: KnowledgeBackend | null
|
||||
}
|
||||
|
||||
async function currentConfig(deps: { data: import("@conusai-kit/data").DataSource }, scope: KnowledgeOrgScope) {
|
||||
const row = (await deps.data.forScope(scope).of(config).list({ limit: 1 })).items[0]
|
||||
return row ?? null
|
||||
}
|
||||
|
||||
export async function getSettings(
|
||||
deps: { data: import("@conusai-kit/data").DataSource },
|
||||
scope: KnowledgeOrgScope,
|
||||
): Promise<SettingsView> {
|
||||
const row = await currentConfig(deps, scope)
|
||||
return { enabled: row?.enabled ?? false, backend: row ? "platform" : null }
|
||||
}
|
||||
|
||||
export interface SettingsDeps {
|
||||
data: import("@conusai-kit/data").DataSource
|
||||
platform: DocumentAiPlatform
|
||||
now?: () => number
|
||||
}
|
||||
|
||||
export async function enableDocumentAi(deps: SettingsDeps, scope: KnowledgeOrgScope): Promise<SettingsView> {
|
||||
const existing = await currentConfig(deps, scope)
|
||||
if (existing?.enabled) return { enabled: true, backend: "platform" }
|
||||
|
||||
const eligibility = await deps.platform.checkEligibility(scope)
|
||||
if (!eligibility.ok) throw enablementDenied(eligibility.reason, eligibility.detail)
|
||||
|
||||
const remaining = await deps.platform.indexingQuota(scope)
|
||||
if (remaining <= 0) throw enablementDenied("quota_exhausted", "the indexing volume for today is spent")
|
||||
|
||||
await deps.data.unitOfWork.run(scope, async (tx) => {
|
||||
const row = (await tx.of(config).list({ limit: 1 })).items[0]
|
||||
const patch = {
|
||||
enabled: true,
|
||||
backend: "platform" as const,
|
||||
source: "managed" as const,
|
||||
serviceId: null,
|
||||
url: null,
|
||||
updatedBy: scope.actor.userId,
|
||||
configuredAt: new Date((deps.now ?? Date.now)()).toISOString(),
|
||||
}
|
||||
if (row) await tx.of(config).update(row.id, patch, row.version)
|
||||
else await tx.of(config).insert(patch)
|
||||
})
|
||||
return { enabled: true, backend: "platform" }
|
||||
}
|
||||
|
||||
export async function disableDocumentAi(
|
||||
deps: Pick<SettingsDeps, "data" | "now">,
|
||||
scope: KnowledgeOrgScope,
|
||||
): Promise<{ enabled: false }> {
|
||||
await deps.data.unitOfWork.run(scope, async (tx) => {
|
||||
const row = (await tx.of(config).list({ limit: 1 })).items[0]
|
||||
if (!row?.enabled) return
|
||||
// Everything else is preserved on purpose: the recorded configuration, every collection and
|
||||
// document, and the service's index. Disabling fences new work; it deletes nothing.
|
||||
await tx.of(config).update(
|
||||
row.id,
|
||||
{
|
||||
enabled: false,
|
||||
updatedBy: scope.actor.userId,
|
||||
configuredAt: new Date((deps.now ?? Date.now)()).toISOString(),
|
||||
},
|
||||
row.version,
|
||||
)
|
||||
})
|
||||
return { enabled: false }
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
// The status a page reads: the module flag, the search service's readiness (separate from the
|
||||
// indexing indicator, which is the per-document rows), and the document counts by progress state.
|
||||
|
||||
import { type DocumentStatus, documents, type KnowledgeRefusal } from "../contract/index.js"
|
||||
import { type KnowledgeDeps, type KnowledgeReadiness, type KnowledgeScope, knowledgeEnabled } from "./ports.js"
|
||||
|
||||
const PAGE = 200
|
||||
|
||||
export interface KnowledgeStatus {
|
||||
enabled: boolean
|
||||
service: {
|
||||
state: "registered" | "indexing" | "ready" | "degraded" | "disabled" | "unavailable"
|
||||
reasons: string[]
|
||||
refusal?: KnowledgeRefusal
|
||||
}
|
||||
documents: Record<DocumentStatus, number>
|
||||
}
|
||||
|
||||
/** What the status read needs; `readiness` is absent in a process that binds no search backend. */
|
||||
export type KnowledgeStatusDeps = Pick<KnowledgeDeps, "data" | "flags"> & {
|
||||
readonly readiness?: KnowledgeReadiness | undefined
|
||||
}
|
||||
|
||||
export async function knowledgeStatusOf(deps: KnowledgeStatusDeps, scope: KnowledgeScope): Promise<KnowledgeStatus> {
|
||||
const counts: Record<DocumentStatus, number> = { queued: 0, indexing: 0, indexed: 0, failed: 0, cancelled: 0 }
|
||||
let cursor: string | undefined
|
||||
for (;;) {
|
||||
const page = await deps.data
|
||||
.forScope(scope)
|
||||
.of(documents)
|
||||
.list({ limit: PAGE, ...(cursor === undefined ? {} : { cursor }) })
|
||||
for (const row of page.items) counts[row.status as DocumentStatus] += 1
|
||||
if (!page.hasMore) break
|
||||
cursor = page.cursor
|
||||
}
|
||||
const enabled = await knowledgeEnabled(deps, scope)
|
||||
const readiness =
|
||||
deps.readiness === undefined
|
||||
? { state: "unavailable" as const, reasons: ["the knowledge search service is not bound in this process"] }
|
||||
: await deps.readiness.readiness(scope).catch((error: unknown) => ({
|
||||
state: "unavailable" as const,
|
||||
reasons: [
|
||||
error instanceof Error
|
||||
? (error.message.split("\n")[0] ?? error.message)
|
||||
: "the search service did not answer",
|
||||
],
|
||||
}))
|
||||
return { enabled, service: readiness, documents: counts }
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
// English messages of the knowledge module. Every string a page of this module shows is a key here
|
||||
// and in lt.ts (spec §10.4); `__tests__/i18n.test.ts` fails when the two differ.
|
||||
|
||||
export const en = {
|
||||
title: "Knowledge",
|
||||
intro:
|
||||
"Search this project's documents by meaning, on the platform's knowledge service. Files are indexed in the background; answers cite the file and the passage they came from.",
|
||||
refusals: {
|
||||
configuration_missing: "Set CONUSAI_ENVIRONMENT for this deployment before using knowledge search.",
|
||||
plan_required: "The organization's plan does not include document search.",
|
||||
feature_disabled: "Document search is switched off for this organization by platform policy.",
|
||||
quota_exhausted: "Today's indexing allowance is spent. Try again tomorrow.",
|
||||
budget_exhausted: "The organization's AI budget is spent. An administrator can raise it on the platform.",
|
||||
scope_missing:
|
||||
"This deployment's token does not hold knowledge.collections:use. Deploy the app again, then have an owner or admin approve it.",
|
||||
indexing: "The service is building its index. Try again in a few minutes.",
|
||||
approval_required: "An administrator must approve this on the platform first.",
|
||||
runtime_unavailable: "The knowledge service did not answer. Try again later.",
|
||||
},
|
||||
failures: {
|
||||
processing_exhausted: "Processing stopped after its final retry. Check services and permissions before retrying.",
|
||||
file_changed: "The source file changed. Upload the original again.",
|
||||
not_text: "The file is not text.",
|
||||
not_indexable: "This kind of file cannot be indexed.",
|
||||
too_large: "The file is too large to index.",
|
||||
},
|
||||
status: {
|
||||
heading: "Service status",
|
||||
flagOff:
|
||||
"Knowledge is switched off. An organization administrator enables document AI in the organization's settings.",
|
||||
state: "Search service",
|
||||
stateReady: "Ready",
|
||||
stateIndexing: "Indexing",
|
||||
stateRegistered: "Registered",
|
||||
stateDegraded: "Degraded",
|
||||
stateDisabled: "Disabled by policy",
|
||||
stateUnavailable: "Unavailable",
|
||||
reasons: "Reasons",
|
||||
indexing: "Indexing in progress",
|
||||
documents: "{queued} queued, {indexing} indexing, {indexed} indexed, {failed} failed",
|
||||
},
|
||||
collections: {
|
||||
heading: "Collections",
|
||||
empty: "No collections yet.",
|
||||
name: "Name",
|
||||
documents: "Documents",
|
||||
indexed: "Indexed",
|
||||
createHeading: "New collection",
|
||||
nameLabel: "Name",
|
||||
nameHint: "Lower-case letters, digits and dashes.",
|
||||
create: "Create collection",
|
||||
creating: "Creating",
|
||||
pickerLabel: "Collection",
|
||||
pick: "Choose a collection",
|
||||
},
|
||||
autoIndex: {
|
||||
heading: "Index uploaded PDFs automatically",
|
||||
hint: "Every PDF uploaded to this project is read and indexed into the chosen collection, with page references. Your permission to manage knowledge is checked again for each file.",
|
||||
label: "Collection",
|
||||
off: "Off",
|
||||
submit: "Save",
|
||||
busy: "Saving…",
|
||||
done: "Saved.",
|
||||
},
|
||||
upload: {
|
||||
heading: "Index a document",
|
||||
label: "File",
|
||||
hint: "Plain text, Markdown, JSON or CSV, up to 1 MB.",
|
||||
submit: "Index",
|
||||
busy: "Indexing requested",
|
||||
done: "The document is queued for indexing.",
|
||||
failed: "That did not work: {detail}",
|
||||
},
|
||||
documents: {
|
||||
heading: "Documents",
|
||||
empty: "Nothing is indexed in this collection yet.",
|
||||
file: "File",
|
||||
status: "Status",
|
||||
statusQueued: "Queued",
|
||||
statusIndexing: "Indexing",
|
||||
statusIndexed: "Indexed",
|
||||
statusFailed: "Failed",
|
||||
statusCancelled: "Removed",
|
||||
error: "Error",
|
||||
remove: "Remove {filename}",
|
||||
removeShort: "Remove",
|
||||
},
|
||||
search: {
|
||||
heading: "Search",
|
||||
label: "Question",
|
||||
placeholder: "Ask across the indexed documents",
|
||||
submit: "Search",
|
||||
busy: "Searching",
|
||||
empty: "No answers in this collection.",
|
||||
results: "{count, plural, one {# answer} other {# answers}}",
|
||||
snippet: "Passage",
|
||||
source: "Source",
|
||||
chunk: "result {chunk}",
|
||||
page: "page {page}",
|
||||
score: "relevance {score}",
|
||||
disclosure: "AI-assisted retrieval: answers are ranked by a model and must be checked against the source.",
|
||||
},
|
||||
settings: {
|
||||
heading: "Document AI",
|
||||
state: "State",
|
||||
stateOn: "Enabled",
|
||||
stateOff: "Disabled",
|
||||
backend: "Backend",
|
||||
backendPlatform: "The platform's knowledge service",
|
||||
enable: "Enable document AI",
|
||||
enabling: "Enabling",
|
||||
disable: "Disable",
|
||||
disabling: "Disabling",
|
||||
enabledDone: "Document AI is enabled.",
|
||||
disabledDone: "Document AI is disabled. Collections, documents and the index are preserved.",
|
||||
planRequired: "The organization's plan does not include document AI.",
|
||||
featureDisabled: "Document AI is disabled by platform policy.",
|
||||
quotaExhausted: "The indexing quota is exhausted; try again later.",
|
||||
upgrade: "Upgrade the plan to enable document AI.",
|
||||
hint: "Enabling checks the plan and the rollout flags and records the choice in this organization; the platform's knowledge service keeps the index. Disabling keeps every collection, document and indexed passage.",
|
||||
},
|
||||
} as const
|
||||
@@ -0,0 +1,2 @@
|
||||
export { en } from "./en.js"
|
||||
export { lt } from "./lt.js"
|
||||
@@ -0,0 +1,126 @@
|
||||
// Lietuviški knowledge modulio pranešimai. Kiekvienas puslapio tekstas yra raktas čia ir en.ts
|
||||
// faile; `__tests__/i18n.test.ts` žlunga, kai jie skiriasi.
|
||||
|
||||
import type { en } from "./en.js"
|
||||
|
||||
type Messages<T> = { [K in keyof T]: T[K] extends string ? string : Messages<T[K]> }
|
||||
|
||||
export const lt: Messages<typeof en> = {
|
||||
title: "Žinios",
|
||||
intro:
|
||||
"Ieškokite šio projekto dokumentų pagal prasmę platformos žinių paslaugoje. Failai indeksuojami fone; atsakymai nurodo failą ir ištrauką, iš kurios paimti.",
|
||||
refusals: {
|
||||
configuration_missing: "Prieš naudodami žinių paiešką šiame diegime nustatykite CONUSAI_ENVIRONMENT.",
|
||||
plan_required: "Organizacijos planas dokumentų paieškos neapima.",
|
||||
feature_disabled: "Dokumentų paiešką šiai organizacijai išjungė platformos politika.",
|
||||
quota_exhausted: "Šios dienos indeksavimo limitas išnaudotas. Bandykite rytoj.",
|
||||
budget_exhausted: "Organizacijos DI biudžetas išnaudotas. Administratorius gali jį padidinti platformoje.",
|
||||
scope_missing:
|
||||
"Šio diegimo prieigos raktas neturi leidimo knowledge.collections:use. Įdiekite programą iš naujo, tada paprašykite savininko ar administratoriaus jį patvirtinti.",
|
||||
indexing: "Paslauga kuria indeksą. Bandykite po kelių minučių.",
|
||||
approval_required: "Pirmiausia tai platformoje turi patvirtinti administratorius.",
|
||||
runtime_unavailable: "Žinių paslauga neatsakė. Bandykite vėliau.",
|
||||
},
|
||||
failures: {
|
||||
processing_exhausted:
|
||||
"Apdorojimas sustabdytas po paskutinio bandymo. Prieš kartodami patikrinkite paslaugas ir teises.",
|
||||
file_changed: "Pasikeitė šaltinio failas. Dar kartą įkelkite originalą.",
|
||||
not_text: "Failas nėra tekstinis.",
|
||||
not_indexable: "Tokio tipo failo indeksuoti negalima.",
|
||||
too_large: "Failas per didelis, kad būtų suindeksuotas.",
|
||||
},
|
||||
status: {
|
||||
heading: "Paslaugos būsena",
|
||||
flagOff: "Žinios išjungtos. Dokumentų DI organizacijos nustatymuose įjungia organizacijos administratorius.",
|
||||
state: "Paieškos paslauga",
|
||||
stateReady: "Pasiruošusi",
|
||||
stateIndexing: "Indeksuoja",
|
||||
stateRegistered: "Registruota",
|
||||
stateDegraded: "Nukentėjusi",
|
||||
stateDisabled: "Išjungta politikos",
|
||||
stateUnavailable: "Neprieinama",
|
||||
reasons: "Priežastys",
|
||||
indexing: "Vyksta indeksavimas",
|
||||
documents: "{queued} laukia, {indexing} indeksuojama, {indexed} suindeksuota, {failed} nepavyko",
|
||||
},
|
||||
collections: {
|
||||
heading: "Kolekcijos",
|
||||
empty: "Kolekcijų dar nėra.",
|
||||
name: "Pavadinimas",
|
||||
documents: "Dokumentai",
|
||||
indexed: "Suindeksuota",
|
||||
createHeading: "Nauja kolekcija",
|
||||
nameLabel: "Pavadinimas",
|
||||
nameHint: "Mažosios raidės, skaičiai ir brūkšneliai.",
|
||||
create: "Sukurti kolekciją",
|
||||
creating: "Kuriama",
|
||||
pickerLabel: "Kolekcija",
|
||||
pick: "Pasirinkite kolekciją",
|
||||
},
|
||||
autoIndex: {
|
||||
heading: "Automatiškai indeksuoti įkeltus PDF",
|
||||
hint: "Kiekvienas į šį projektą įkeltas PDF perskaitomas ir indeksuojamas pasirinktoje kolekcijoje su puslapių nuorodomis. Jūsų teisė tvarkyti žinias tikrinama iš naujo kiekvienam failui.",
|
||||
label: "Kolekcija",
|
||||
off: "Išjungta",
|
||||
submit: "Išsaugoti",
|
||||
busy: "Išsaugoma…",
|
||||
done: "Išsaugota.",
|
||||
},
|
||||
upload: {
|
||||
heading: "Indeksuoti dokumentą",
|
||||
label: "Failas",
|
||||
hint: "Grynasis tekstas, Markdown, JSON arba CSV, iki 1 MB.",
|
||||
submit: "Indeksuoti",
|
||||
busy: "Indeksavimas paprašytas",
|
||||
done: "Dokumentas eilėje indeksavimui.",
|
||||
failed: "Nepavyko: {detail}",
|
||||
},
|
||||
documents: {
|
||||
heading: "Dokumentai",
|
||||
empty: "Šioje kolekcijoje dar nieko suindeksuota.",
|
||||
file: "Failas",
|
||||
status: "Būsena",
|
||||
statusQueued: "Laukia",
|
||||
statusIndexing: "Indeksuojama",
|
||||
statusIndexed: "Suindeksuota",
|
||||
statusFailed: "Nepavyko",
|
||||
statusCancelled: "Pašalinta",
|
||||
error: "Klaida",
|
||||
remove: "Pašalinti: {filename}",
|
||||
removeShort: "Pašalinti",
|
||||
},
|
||||
search: {
|
||||
heading: "Paieška",
|
||||
label: "Klausimas",
|
||||
placeholder: "Klauskite per suindeksuotus dokumentus",
|
||||
submit: "Ieškoti",
|
||||
busy: "Ieškoma",
|
||||
empty: "Šioje kolekcijoje atsakymų nėra.",
|
||||
results: "{count, plural, one {# atsakymas} few {# atsakymai} many {# atsakymo} other {# atsakymų}}",
|
||||
snippet: "Ištrauka",
|
||||
source: "Šaltinis",
|
||||
chunk: "{chunk} rezultatas",
|
||||
page: "{page} puslapis",
|
||||
score: "atitiktis {score}",
|
||||
disclosure: "Paieška su DI pagalba: atsakymus rikiuoja modelis, juos būtina patikrinti pagal šaltinį.",
|
||||
},
|
||||
settings: {
|
||||
heading: "Dokumentų DI",
|
||||
state: "Būsena",
|
||||
stateOn: "Įjungta",
|
||||
stateOff: "Išjungta",
|
||||
backend: "Backend'as",
|
||||
backendPlatform: "Platformos žinių paslauga",
|
||||
enable: "Įjungti dokumentų DI",
|
||||
enabling: "Įjungiama",
|
||||
disable: "Išjungti",
|
||||
disabling: "Išjungiama",
|
||||
enabledDone: "Dokumentų DI įjungta.",
|
||||
disabledDone: "Dokumentų DI išjungta. Kolekcijos, dokumentai ir indeksas išsaugoti.",
|
||||
planRequired: "Organizacijos planas dokumentų DI neapima.",
|
||||
featureDisabled: "Dokumentų DI išjungta platformos politikos.",
|
||||
quotaExhausted: "Indeksavimo kvota išsemta; bandykite vėliau.",
|
||||
upgrade: "Atnaujinkite planą, kad įjungtumėte dokumentų DI.",
|
||||
hint: "Įjungdama tikrina planą ir išleidimo požymius ir įrašo pasirinkimą šioje organizacijoje; indeksą saugo platformos žinių paslauga. Išjungdama viską išsaugo.",
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
// The knowledge module's public surface: the manifest and what the composition roots bind.
|
||||
|
||||
export type { SdkDocumentAiPlatformOptions } from "./adapters/enablement.js"
|
||||
export { AGENT_DOCUMENT_TOOLS_FLAG, AGENT_KNOWLEDGE_TOOLS_FLAG, sdkDocumentAiPlatform } from "./adapters/enablement.js"
|
||||
export { platformKnowledgeSearch, platformPassageIndex } from "./adapters/platform-knowledge.js"
|
||||
export type { SdkKnowledgeReaderOptions } from "./adapters/reader.js"
|
||||
export { sdkKnowledgeReader } from "./adapters/reader.js"
|
||||
export { platformKnowledgeReadiness } from "./adapters/readiness.js"
|
||||
export * from "./contract/index.js"
|
||||
export type {
|
||||
DocumentAiPlatform,
|
||||
KnowledgeDeps,
|
||||
KnowledgeFlags,
|
||||
KnowledgeHit,
|
||||
KnowledgeOrgScope,
|
||||
KnowledgePassage,
|
||||
KnowledgePassageIndex,
|
||||
KnowledgeReader,
|
||||
KnowledgeReadiness,
|
||||
KnowledgeScope,
|
||||
KnowledgeSearchService,
|
||||
} from "./core/ports.js"
|
||||
export {
|
||||
KNOWLEDGE_ENABLEMENT,
|
||||
KNOWLEDGE_FLAGS,
|
||||
KNOWLEDGE_PASSAGES,
|
||||
KNOWLEDGE_READER,
|
||||
KNOWLEDGE_READINESS,
|
||||
KNOWLEDGE_SCOPE,
|
||||
KNOWLEDGE_SEARCH,
|
||||
} from "./core/ports.js"
|
||||
export { convexKnowledgeModule } from "./module.convex.js"
|
||||
export { knowledgeModule } from "./module.js"
|
||||
export { KNOWLEDGE_MODULE_BASE } from "./module.shared.js"
|
||||
@@ -0,0 +1,68 @@
|
||||
-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0
|
||||
create table "knowledge_collection" (
|
||||
"id" uuid primary key,
|
||||
"version" integer not null,
|
||||
"created_at" timestamptz not null,
|
||||
"updated_at" timestamptz not null,
|
||||
"org_id" uuid not null,
|
||||
"workspace_id" uuid not null,
|
||||
"project_id" uuid not null,
|
||||
"name" text not null,
|
||||
"created_by" text not null
|
||||
);
|
||||
|
||||
alter table "knowledge_collection" add constraint "knowledge_collection_org_id_workspace_id_project_id_name_key" unique nulls not distinct ("org_id", "workspace_id", "project_id", "name");
|
||||
|
||||
create table "knowledge_config" (
|
||||
"id" uuid primary key,
|
||||
"version" integer not null,
|
||||
"created_at" timestamptz not null,
|
||||
"updated_at" timestamptz not null,
|
||||
"org_id" uuid not null,
|
||||
"enabled" boolean not null,
|
||||
"backend" text not null,
|
||||
"source" text not null,
|
||||
"service_id" text,
|
||||
"url" text,
|
||||
"updated_by" text not null,
|
||||
"configured_at" text not null
|
||||
);
|
||||
|
||||
create table "knowledge_document" (
|
||||
"id" uuid primary key,
|
||||
"version" integer not null,
|
||||
"created_at" timestamptz not null,
|
||||
"updated_at" timestamptz not null,
|
||||
"org_id" uuid not null,
|
||||
"workspace_id" uuid not null,
|
||||
"project_id" uuid not null,
|
||||
"collection_name" text not null,
|
||||
"file_id" uuid not null,
|
||||
"filename" text not null,
|
||||
"status" text not null,
|
||||
"error" text,
|
||||
"indexed_at" text,
|
||||
"requested_by" text not null
|
||||
);
|
||||
|
||||
alter table "knowledge_document" add constraint "knowledge_document_org_id_workspace_id_project_id_12648b24_key" unique nulls not distinct ("org_id", "workspace_id", "project_id", "collection_name", "file_id");
|
||||
|
||||
create index "knowledge_document_org_id_workspace_id_project_id_ec03a581_idx" on "knowledge_document" ("org_id", "workspace_id", "project_id", "collection_name");
|
||||
|
||||
alter table "knowledge_collection" enable row level security;
|
||||
|
||||
alter table "knowledge_collection" force row level security;
|
||||
|
||||
create policy "knowledge_collection_tenant" on "knowledge_collection" using ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid) with check ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid);
|
||||
|
||||
alter table "knowledge_config" enable row level security;
|
||||
|
||||
alter table "knowledge_config" force row level security;
|
||||
|
||||
create policy "knowledge_config_tenant" on "knowledge_config" using ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid) with check ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid);
|
||||
|
||||
alter table "knowledge_document" enable row level security;
|
||||
|
||||
alter table "knowledge_document" force row level security;
|
||||
|
||||
create policy "knowledge_document_tenant" on "knowledge_document" using ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid) with check ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid);
|
||||
@@ -0,0 +1,19 @@
|
||||
-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0
|
||||
create table "knowledge_auto_index" (
|
||||
"id" uuid primary key,
|
||||
"version" integer not null,
|
||||
"created_at" timestamptz not null,
|
||||
"updated_at" timestamptz not null,
|
||||
"org_id" uuid not null,
|
||||
"workspace_id" uuid not null,
|
||||
"project_id" uuid not null,
|
||||
"collection_name" text not null,
|
||||
"enabled_by" text not null,
|
||||
"enabled_at" text not null
|
||||
);
|
||||
|
||||
alter table "knowledge_auto_index" enable row level security;
|
||||
|
||||
alter table "knowledge_auto_index" force row level security;
|
||||
|
||||
create policy "knowledge_auto_index_tenant" on "knowledge_auto_index" using ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid) with check ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid);
|
||||
@@ -0,0 +1,22 @@
|
||||
-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0
|
||||
create table "knowledge_passage_set" (
|
||||
"id" uuid primary key,
|
||||
"version" integer not null,
|
||||
"created_at" timestamptz not null,
|
||||
"updated_at" timestamptz not null,
|
||||
"org_id" uuid not null,
|
||||
"workspace_id" uuid not null,
|
||||
"project_id" uuid not null,
|
||||
"collection_name" text not null,
|
||||
"file_id" uuid not null,
|
||||
"document_ref" text not null,
|
||||
"pages" text not null
|
||||
);
|
||||
|
||||
alter table "knowledge_passage_set" add constraint "knowledge_passage_set_org_id_workspace_id_project_id_collection_name_file_id_key" unique nulls not distinct ("org_id", "workspace_id", "project_id", "collection_name", "file_id");
|
||||
|
||||
alter table "knowledge_passage_set" enable row level security;
|
||||
|
||||
alter table "knowledge_passage_set" force row level security;
|
||||
|
||||
create policy "knowledge_passage_set_tenant" on "knowledge_passage_set" using ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid) with check ("org_id" = nullif(current_setting('app.org_id', true), '')::uuid);
|
||||
@@ -0,0 +1,24 @@
|
||||
// The Convex profile binding of the knowledge module, shipped fail-closed for a future wiring:
|
||||
// the integration review kept the module Postgres-only because the search effect, the
|
||||
// index/remove jobs and the enablement flow all leave the process and have no composed Convex
|
||||
// Node actions yet (see README.md). Knowledge indexing needs the Node runtime boundary (the platform's
|
||||
// knowledge client), exactly like documents' OCR: the job handlers here fail
|
||||
// closed so the work can only reach the platform SDKs through a composed Node action, never the
|
||||
// transaction dispatcher.
|
||||
|
||||
import { defineModule, internal, type JobDef } from "@conusai-kit/core"
|
||||
import { type DocumentJobPayload, documentPayload } from "./adapters/jobs.js"
|
||||
import { INDEX_JOB, REMOVE_JOB } from "./contract/index.js"
|
||||
import { KNOWLEDGE_MODULE_BASE } from "./module.shared.js"
|
||||
|
||||
const actionOnly = (name: string): JobDef<DocumentJobPayload> => ({
|
||||
payload: documentPayload,
|
||||
async handler() {
|
||||
throw internal(`${name} must run through the Convex Node action boundary`)
|
||||
},
|
||||
})
|
||||
|
||||
export const convexKnowledgeModule = defineModule({
|
||||
...KNOWLEDGE_MODULE_BASE,
|
||||
jobs: { [INDEX_JOB]: actionOnly(INDEX_JOB), [REMOVE_JOB]: actionOnly(REMOVE_JOB) },
|
||||
})
|
||||
@@ -0,0 +1,98 @@
|
||||
// Shared, runtime-neutral knowledge manifest data. The normal app binds its worker implementations
|
||||
// of the index/remove jobs; the Convex binding file (module.convex.ts) ships fail-closed for a
|
||||
// future wiring — the module stays Postgres-only until the search effect and the jobs have their
|
||||
// composed Node actions (see README.md).
|
||||
|
||||
import { contribute, type ModuleManifest } from "@conusai-kit/core"
|
||||
import { repositoryBindings } from "@conusai-kit/data"
|
||||
import { filesProcessors } from "@conusai-kit/files"
|
||||
import { accountData } from "@conusai-kit/tenancy"
|
||||
import { knowledgeAccountData } from "./adapters/account.js"
|
||||
import { KNOWLEDGE_PROCEDURE_IMPLS } from "./adapters/procedures.js"
|
||||
import {
|
||||
AUTO_INDEX_CHANGED,
|
||||
AUTO_INDEX_CONTENT_TYPES,
|
||||
AUTO_INDEX_JOB,
|
||||
AUTO_INDEX_PROCESSOR_KEY,
|
||||
COLLECTION_CREATED,
|
||||
DOCUMENT_AI_DISABLED,
|
||||
DOCUMENT_AI_ENABLED,
|
||||
INDEX_JOB,
|
||||
KNOWLEDGE_COLLECTIONS,
|
||||
KNOWLEDGE_ENTITIES,
|
||||
KNOWLEDGE_FLAG_DEFAULTS,
|
||||
} from "./contract/index.js"
|
||||
import { en, lt } from "./i18n/index.js"
|
||||
|
||||
const tables = repositoryBindings(...KNOWLEDGE_ENTITIES)
|
||||
|
||||
export const KNOWLEDGE_MODULE_BASE: ModuleManifest = {
|
||||
id: "knowledge",
|
||||
version: "0.1.0",
|
||||
kit: "^0.1.0",
|
||||
requires: ["tenancy", "files"],
|
||||
serviceRequirements: [
|
||||
{
|
||||
transport: "platform",
|
||||
service: "knowledge",
|
||||
apiRange: "=1.0.0",
|
||||
profiles: ["postgres"],
|
||||
operations: {
|
||||
[INDEX_JOB]: ["knowledge.collections:use"],
|
||||
[AUTO_INDEX_JOB]: ["knowledge.collections:use"],
|
||||
"knowledge.search": ["knowledge.collections:use"],
|
||||
},
|
||||
configuration: ["ai_processing"],
|
||||
dependencies: ["blob", "ai"],
|
||||
dataClasses: ["document", "personal", "ai-input"],
|
||||
partitionRequired: false,
|
||||
remediation: "/settings/services/knowledge",
|
||||
},
|
||||
{
|
||||
// The automatic PDF intake reads an uploaded PDF through the platform's document reader (core-11).
|
||||
transport: "platform",
|
||||
service: "ocr",
|
||||
apiRange: "=1.0.0-preview.20261103",
|
||||
profiles: ["postgres"],
|
||||
operations: { [AUTO_INDEX_JOB]: ["ocr:use"] },
|
||||
configuration: ["ocr_processing"],
|
||||
dependencies: ["blob"],
|
||||
dataClasses: ["document", "personal"],
|
||||
partitionRequired: false,
|
||||
remediation: "/settings/services/ocr",
|
||||
},
|
||||
],
|
||||
permissions: {
|
||||
// Searching and reading the status is a member's act; the kit's member maps onto "editor".
|
||||
"knowledge.knowledge.search": "editor",
|
||||
// Managing collections, indexing and removing documents, and the enablement flow are
|
||||
// administrative acts.
|
||||
"knowledge.collection.manage": "admin",
|
||||
"knowledge.knowledge.manage": "admin",
|
||||
"knowledge.settings.manage": "admin",
|
||||
},
|
||||
procedures: KNOWLEDGE_PROCEDURE_IMPLS,
|
||||
// Indexing is live (TS6.1): a change of knowledge_document reaches every member's open page.
|
||||
collections: [...KNOWLEDGE_COLLECTIONS],
|
||||
repositories: { postgres: tables },
|
||||
migrations: { postgres: "migrations" },
|
||||
// The module ships the Postgres profile; the Convex binding file (module.convex.ts) ships
|
||||
// fail-closed for a future wiring and is deliberately NOT registered in convex/ — the search
|
||||
// effect, the jobs and the enablement flow all leave the process and have no composed Convex
|
||||
// Node actions yet (see README.md).
|
||||
profiles: ["postgres"],
|
||||
audit: [COLLECTION_CREATED, DOCUMENT_AI_ENABLED, DOCUMENT_AI_DISABLED, AUTO_INDEX_CHANGED],
|
||||
flags: Object.fromEntries(Object.entries(KNOWLEDGE_FLAG_DEFAULTS).map(([flag, value]) => [flag, { default: value }])),
|
||||
ui: {
|
||||
nav: [{ scope: "project", label: "knowledge.title", href: "knowledge", permission: "knowledge.knowledge.search" }],
|
||||
},
|
||||
i18n: { en, lt },
|
||||
// An opted-in project's uploaded PDFs, indexed automatically (core-11): the job reads nothing for a
|
||||
// project that has not opted in.
|
||||
extends: [
|
||||
contribute(accountData, [knowledgeAccountData]),
|
||||
contribute(filesProcessors, [
|
||||
{ key: AUTO_INDEX_PROCESSOR_KEY, contentTypes: [...AUTO_INDEX_CONTENT_TYPES], job: AUTO_INDEX_JOB },
|
||||
]),
|
||||
],
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
// The knowledge module's manifest (spec §4.1): the one place it registers anything. The normal
|
||||
// (Postgres) app binds the worker implementations of the index/remove jobs; the module's own
|
||||
// outbox subscribers enqueue them, so a web request writes the row and answers, never waiting on
|
||||
// the platform's knowledge service.
|
||||
|
||||
import { defineModule } from "@conusai-kit/core"
|
||||
import { KNOWLEDGE_JOBS, KNOWLEDGE_SUBSCRIPTIONS } from "./adapters/jobs.js"
|
||||
import { KNOWLEDGE_EVENTS } from "./contract/index.js"
|
||||
import { KNOWLEDGE_MODULE_BASE } from "./module.shared.js"
|
||||
|
||||
export const knowledgeModule = defineModule({
|
||||
...KNOWLEDGE_MODULE_BASE,
|
||||
jobs: KNOWLEDGE_JOBS,
|
||||
events: { publishes: [...KNOWLEDGE_EVENTS], subscribes: KNOWLEDGE_SUBSCRIPTIONS },
|
||||
})
|
||||
@@ -0,0 +1,66 @@
|
||||
"use client"
|
||||
|
||||
// The project's automatic PDF indexing (core-11): an administrator picks the collection uploaded PDFs
|
||||
// go into, or turns it off. The write answers the new state, so the form shows it without re-running
|
||||
// the page's loader (the module declares live collections; test/live-writes.test.ts).
|
||||
|
||||
import type { ProjectSlugs } from "@conusai-kit/ui"
|
||||
import { useWrite } from "@conusai-kit/ui/data"
|
||||
import { FormLayout, PageSection, SelectField } from "@conusai-kit/ui/patterns"
|
||||
import { type ReactElement, useState } from "react"
|
||||
|
||||
export interface AutoIndexFormLabels {
|
||||
readonly heading: string
|
||||
readonly hint: string
|
||||
readonly label: string
|
||||
readonly off: string
|
||||
readonly submit: string
|
||||
readonly busy: string
|
||||
readonly done: string
|
||||
}
|
||||
|
||||
/** The select's value for "off": a collection name is lower-case letters, digits and dashes, never this. */
|
||||
const OFF = "_off"
|
||||
|
||||
export function AutoIndexForm({
|
||||
slugs,
|
||||
collections,
|
||||
current,
|
||||
labels,
|
||||
}: {
|
||||
slugs: ProjectSlugs
|
||||
collections: readonly string[]
|
||||
current: string | null
|
||||
labels: AutoIndexFormLabels
|
||||
}): ReactElement {
|
||||
const setAutoIndex = useWrite<ProjectSlugs & { collectionName: string | null }, { collectionName: string | null }>(
|
||||
"knowledge.autoindex.set",
|
||||
{ org: slugs.org },
|
||||
)
|
||||
const [selected, setSelected] = useState<string | null>(current)
|
||||
const [done, setDone] = useState(false)
|
||||
|
||||
const save = async (_: unknown, data: FormData): Promise<void> => {
|
||||
const value = data.get("collection")
|
||||
const collectionName = typeof value === "string" && value !== OFF ? value : null
|
||||
setDone(false)
|
||||
const answer = await setAutoIndex.mutate({ ...slugs, collectionName })
|
||||
setSelected(answer.collectionName)
|
||||
setDone(true)
|
||||
}
|
||||
|
||||
return (
|
||||
<PageSection title={labels.heading} level={3} variant="card">
|
||||
<p>{labels.hint}</p>
|
||||
<FormLayout submitLabel={labels.submit} busyLabel={labels.busy} onSubmit={save}>
|
||||
<SelectField
|
||||
name="collection"
|
||||
label={labels.label}
|
||||
defaultValue={selected ?? OFF}
|
||||
options={[{ value: OFF, label: labels.off }, ...collections.map((name) => ({ value: name, label: name }))]}
|
||||
/>
|
||||
</FormLayout>
|
||||
{done ? <p role="status">{labels.done}</p> : null}
|
||||
</PageSection>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
"use client"
|
||||
|
||||
// Upload and index: the browser sends the file to the files module (the kit's `uploadFile`), then
|
||||
// asks this module to index it (`knowledge.document.index`). The indexing itself is a worker job;
|
||||
// this form only ever waits on the request, which resolves once the live stream has seen the document's
|
||||
// row, so the collection's live list already shows it queued.
|
||||
|
||||
import type { ProjectSlugs } from "@conusai-kit/ui"
|
||||
import { useUpload, useWrite } from "@conusai-kit/ui/data"
|
||||
import { FormLayout, PageSection, TextField } from "@conusai-kit/ui/patterns"
|
||||
import { type ReactElement, useState } from "react"
|
||||
import { INDEXABLE_CONTENT_TYPES, MAX_INDEX_BYTES } from "../contract/index.js"
|
||||
|
||||
export interface IndexFormLabels {
|
||||
readonly heading: string
|
||||
readonly label: string
|
||||
readonly hint: string
|
||||
readonly submit: string
|
||||
readonly busy: string
|
||||
readonly done: string
|
||||
}
|
||||
|
||||
export function IndexForm({
|
||||
slugs,
|
||||
collectionName,
|
||||
labels,
|
||||
}: {
|
||||
slugs: ProjectSlugs
|
||||
collectionName: string
|
||||
labels: IndexFormLabels
|
||||
}): ReactElement {
|
||||
const upload = useUpload()
|
||||
const indexDocument = useWrite<ProjectSlugs & { collectionName: string; fileId: string }, unknown>(
|
||||
"knowledge.document.index",
|
||||
{ org: slugs.org },
|
||||
)
|
||||
const [done, setDone] = useState(false)
|
||||
|
||||
const index = async (_: unknown, data: FormData): Promise<void> => {
|
||||
const file = data.get("file")
|
||||
if (!(file instanceof File) || file.size === 0) return
|
||||
setDone(false)
|
||||
const stored = await upload(slugs, file)
|
||||
await indexDocument.mutate({ ...slugs, collectionName, fileId: stored.fileId })
|
||||
setDone(true)
|
||||
}
|
||||
|
||||
return (
|
||||
<PageSection title={labels.heading} level={3} variant="card">
|
||||
<FormLayout submitLabel={labels.submit} busyLabel={labels.busy} onSubmit={index}>
|
||||
<TextField
|
||||
name="file"
|
||||
type="file"
|
||||
label={labels.label}
|
||||
description={`${labels.hint} · ${Math.round(MAX_INDEX_BYTES / 1024)} KB.`}
|
||||
accept={INDEXABLE_CONTENT_TYPES.join(",")}
|
||||
required
|
||||
/>
|
||||
</FormLayout>
|
||||
{done ? <p role="status">{labels.done}</p> : null}
|
||||
</PageSection>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
export type { IndexFormLabels } from "./index-form.js"
|
||||
export { IndexForm } from "./index-form.js"
|
||||
export type { SearchFormLabels, StatusLabels } from "./labels.js"
|
||||
export { CollectionDocuments, type DocumentListLabels, useLiveDocuments } from "./live-documents.js"
|
||||
export { knowledgePage } from "./pages.js"
|
||||
export { SearchForm } from "./search-form.js"
|
||||
export type { SettingsSectionLabels } from "./settings-section.js"
|
||||
export { SettingsSection } from "./settings-section.js"
|
||||
@@ -0,0 +1,118 @@
|
||||
// The labels each component takes, built from a message lookup. Every key this module's pages use
|
||||
// is named here and nowhere else, so `__tests__/i18n.test.ts` can run these builders against the
|
||||
// catalogues and fail on a key either language lacks.
|
||||
|
||||
import { KNOWLEDGE_REFUSALS, type KnowledgeRefusal } from "../contract/index.js"
|
||||
|
||||
export type Translate = (key: string, values?: Record<string, string | number>) => string
|
||||
|
||||
/** A typed refusal as a page or a form shows it. */
|
||||
export interface RefusalNotice {
|
||||
readonly code: KnowledgeRefusal
|
||||
readonly text: string
|
||||
}
|
||||
|
||||
export type RefusalLabels = Readonly<Record<KnowledgeRefusal, string>>
|
||||
|
||||
export const refusalLabels = (t: Translate): RefusalLabels =>
|
||||
Object.fromEntries(KNOWLEDGE_REFUSALS.map((code) => [code, t(`refusals.${code}`)])) as RefusalLabels
|
||||
|
||||
/** The typed refusal a procedure answered, or undefined for any other error (which the form words itself). */
|
||||
export function refusalOf(labels: RefusalLabels, error: unknown): RefusalNotice | undefined {
|
||||
const code = (error as { problem?: { code?: unknown } } | null)?.problem?.code
|
||||
return typeof code === "string" && code in labels
|
||||
? { code: code as KnowledgeRefusal, text: labels[code as KnowledgeRefusal] }
|
||||
: undefined
|
||||
}
|
||||
|
||||
/** The words for a failure a document row records: the file's own, or the service's refusal; else the code. */
|
||||
export function failureText(t: Translate, code: string): string {
|
||||
return failureLabels(t)[code] ?? code
|
||||
}
|
||||
|
||||
/** The failures a document row can name with words of their own: the file's, and the service's refusals. */
|
||||
const FILE_FAILURES = ["file_changed", "not_text", "not_indexable", "too_large"] as const
|
||||
|
||||
/** The words of each failure `failureText` knows, by code, for a page that words a live row in the browser. */
|
||||
export const failureLabels = (t: Translate): Readonly<Record<string, string>> =>
|
||||
Object.fromEntries([
|
||||
...KNOWLEDGE_REFUSALS.map((code) => [code, t(`refusals.${code}`)]),
|
||||
...FILE_FAILURES.map((code) => [code, t(`failures.${code}`)]),
|
||||
])
|
||||
|
||||
/** The status shape the labels render, declared here: ui reads no core/. */
|
||||
interface StatusCounts {
|
||||
documents: { queued: number; indexing: number; indexed: number; failed: number; cancelled: number }
|
||||
}
|
||||
|
||||
export interface StatusLabels {
|
||||
readonly heading: string
|
||||
readonly flagOff: string
|
||||
readonly state: string
|
||||
readonly reasons: string
|
||||
readonly indexing: string
|
||||
readonly documents: string
|
||||
}
|
||||
|
||||
export function statusLabels(t: Translate, status: StatusCounts): StatusLabels {
|
||||
return {
|
||||
heading: t("status.heading"),
|
||||
flagOff: t("status.flagOff"),
|
||||
state: t("status.state"),
|
||||
reasons: t("status.reasons"),
|
||||
indexing: t("status.indexing"),
|
||||
documents: t("status.documents", status.documents),
|
||||
}
|
||||
}
|
||||
|
||||
export const serviceStateLabel = (t: Translate, state: string): string => {
|
||||
switch (state) {
|
||||
case "ready":
|
||||
return t("status.stateReady")
|
||||
case "indexing":
|
||||
return t("status.stateIndexing")
|
||||
case "registered":
|
||||
return t("status.stateRegistered")
|
||||
case "degraded":
|
||||
return t("status.stateDegraded")
|
||||
case "disabled":
|
||||
return t("status.stateDisabled")
|
||||
default:
|
||||
return t("status.stateUnavailable")
|
||||
}
|
||||
}
|
||||
|
||||
export interface SearchFormLabels {
|
||||
readonly heading: string
|
||||
readonly label: string
|
||||
readonly placeholder: string
|
||||
readonly submit: string
|
||||
readonly busy: string
|
||||
readonly empty: string
|
||||
readonly results: string
|
||||
readonly snippet: string
|
||||
readonly source: string
|
||||
readonly chunk: string
|
||||
readonly page: string
|
||||
readonly disclosure: string
|
||||
readonly refusals: RefusalLabels
|
||||
}
|
||||
|
||||
export function searchFormLabels(t: Translate): SearchFormLabels {
|
||||
return {
|
||||
heading: t("search.heading"),
|
||||
label: t("search.label"),
|
||||
placeholder: t("search.placeholder"),
|
||||
submit: t("search.submit"),
|
||||
busy: t("search.busy"),
|
||||
empty: t("search.empty"),
|
||||
// Not filled in here: only the browser knows the count, and `formatMessage` brings the plural rules.
|
||||
results: t("search.results"),
|
||||
snippet: t("search.snippet"),
|
||||
source: t("search.source"),
|
||||
chunk: t("search.chunk", { chunk: "{chunk}" }),
|
||||
page: t("search.page", { page: "{page}" }),
|
||||
disclosure: t("search.disclosure"),
|
||||
refusals: refusalLabels(t),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
"use client"
|
||||
|
||||
// One collection's documents, live (umbrella TS6.4 and TS6.6): the page's view renders them and hands them
|
||||
// over as the seed, so the page arrives complete; this keeps them current through the organization's live
|
||||
// stream -- a document another member asks to index appears, and each moves from queued to indexed or
|
||||
// failed as the worker and the platform answer, without a reload. The rows come from
|
||||
// `knowledge.document.getMany`, under the same permission as the list; a topic is the whole project, so the
|
||||
// rows of the project's other collections are filtered out here.
|
||||
|
||||
import type { ProjectSlugs } from "@conusai-kit/ui"
|
||||
import { eq, liveCollection, useDataClient, useLive } from "@conusai-kit/ui/data"
|
||||
import { type ResourceColumn, ResourceList } from "@conusai-kit/ui/patterns"
|
||||
import { type ReactElement, useState } from "react"
|
||||
import { DOCUMENT_COLLECTION, type DocumentView } from "../contract/index.js"
|
||||
|
||||
export interface DocumentListLabels {
|
||||
readonly caption: string
|
||||
readonly file: string
|
||||
readonly status: string
|
||||
readonly error: string
|
||||
/** The message of each document status, by status. */
|
||||
readonly statuses: Readonly<Record<DocumentView["status"], string>>
|
||||
/** The words of each failure code a row can carry; an unknown code is shown as it is. */
|
||||
readonly failures: Readonly<Record<string, string>>
|
||||
}
|
||||
|
||||
export function useLiveDocuments(
|
||||
slugs: ProjectSlugs,
|
||||
collectionName: string,
|
||||
seed: readonly DocumentView[],
|
||||
): readonly DocumentView[] {
|
||||
const client = useDataClient()
|
||||
// One collection per knowledge collection for the life of the page; the seed is the first render's.
|
||||
const [documents] = useState(() =>
|
||||
liveCollection<DocumentView>({
|
||||
client,
|
||||
org: slugs.org,
|
||||
topic: { ws: slugs.workspace, project: slugs.project },
|
||||
entity: DOCUMENT_COLLECTION.entity,
|
||||
key: DOCUMENT_COLLECTION.key,
|
||||
id: `${DOCUMENT_COLLECTION.entity}:${slugs.org}:${slugs.workspace}:${slugs.project}:${collectionName}`,
|
||||
seed,
|
||||
list: {
|
||||
procedure: DOCUMENT_COLLECTION.list,
|
||||
input: { ...slugs, collectionName },
|
||||
rows: (out) => (out as { documents: DocumentView[] }).documents,
|
||||
},
|
||||
getMany: {
|
||||
procedure: DOCUMENT_COLLECTION.getMany,
|
||||
input: { ...slugs },
|
||||
rows: (out) => (out as { items: DocumentView[] }).items,
|
||||
},
|
||||
}),
|
||||
)
|
||||
const { data, isReady } = useLive(documents, (q, rows) =>
|
||||
q
|
||||
.from({ document: rows })
|
||||
.where(({ document }: { document: DocumentView }) => eq(document.collectionName, collectionName)),
|
||||
)
|
||||
// Until the collection is ready the server's rows are the answer, so the first paint matches it.
|
||||
return isReady ? data : seed
|
||||
}
|
||||
|
||||
export function CollectionDocuments({
|
||||
slugs,
|
||||
collectionName,
|
||||
seed,
|
||||
labels,
|
||||
}: {
|
||||
slugs: ProjectSlugs
|
||||
collectionName: string
|
||||
seed: readonly DocumentView[]
|
||||
labels: DocumentListLabels
|
||||
}): ReactElement {
|
||||
const rows = useLiveDocuments(slugs, collectionName, seed)
|
||||
const columns: ResourceColumn<DocumentView>[] = [
|
||||
{ id: "filename", header: labels.file, wrap: true },
|
||||
{ id: "status", header: labels.status, cell: (document) => labels.statuses[document.status] },
|
||||
{
|
||||
id: "error",
|
||||
header: labels.error,
|
||||
wrap: true,
|
||||
cell: (document) => (document.error === null ? "" : (labels.failures[document.error] ?? document.error)),
|
||||
},
|
||||
]
|
||||
return (
|
||||
<ResourceList
|
||||
label={labels.caption}
|
||||
columns={columns}
|
||||
data={rows}
|
||||
rowId={(document) => document.documentId}
|
||||
rowLabel={(document) => document.filename}
|
||||
empty={{ title: labels.caption }}
|
||||
/>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,207 @@
|
||||
// The knowledge module's page, on the shell's F10 contract (docs/reference/app-kit.md): a loader the web
|
||||
// shell runs on the server under `/<org>/<workspace>/<project>/knowledge`, and the view of what it
|
||||
// returned. Every read is a procedure through the same caller the API uses, and every label is translated
|
||||
// in the loader; every write is a client component that calls a procedure through the shell's transport.
|
||||
//
|
||||
// The page renders three things separately on purpose: the service's health (the module flag and
|
||||
// the search service's readiness), the indexing indicator (the per-document progress states), and
|
||||
// the search itself — "indexing" is not "down", and the page never conflates them.
|
||||
|
||||
import { type Container, notFound, type TransportCaller } from "@conusai-kit/core"
|
||||
import type { I18n } from "@conusai-kit/i18n"
|
||||
import { type ProjectPage, type ProjectPageViewProps, projectPage } from "@conusai-kit/ui"
|
||||
import { FactList, Page, PageHeader, PageSection } from "@conusai-kit/ui/patterns"
|
||||
import { Alert, AlertDescription } from "@conusai-kit/ui/primitives"
|
||||
import type { ReactElement } from "react"
|
||||
import {
|
||||
autoIndexGet,
|
||||
type CollectionView,
|
||||
collectionList,
|
||||
type DocumentView,
|
||||
documentList,
|
||||
knowledgeStatus,
|
||||
} from "../contract/index.js"
|
||||
import { AutoIndexForm, type AutoIndexFormLabels } from "./auto-index-form.js"
|
||||
import { IndexForm, type IndexFormLabels } from "./index-form.js"
|
||||
import {
|
||||
failureLabels,
|
||||
type RefusalNotice,
|
||||
refusalLabels,
|
||||
type SearchFormLabels,
|
||||
type StatusLabels,
|
||||
searchFormLabels,
|
||||
serviceStateLabel,
|
||||
statusLabels,
|
||||
} from "./labels.js"
|
||||
import { CollectionDocuments, type DocumentListLabels } from "./live-documents.js"
|
||||
import { SearchForm } from "./search-form.js"
|
||||
|
||||
/** One collection and its documents: the seed of its live list. */
|
||||
export interface KnowledgeCollectionRow {
|
||||
readonly name: string
|
||||
readonly documents: readonly DocumentView[]
|
||||
}
|
||||
|
||||
export interface KnowledgePageData {
|
||||
readonly title: string
|
||||
readonly intro: string
|
||||
readonly status: {
|
||||
readonly enabled: boolean
|
||||
readonly labels: StatusLabels
|
||||
/** The service's state in the reader's language. */
|
||||
readonly state: string
|
||||
readonly reasons: readonly string[]
|
||||
/** Why the service will not serve this deployment, when that is the answer. */
|
||||
readonly refusal?: RefusalNotice
|
||||
}
|
||||
readonly collections: readonly KnowledgeCollectionRow[]
|
||||
readonly documents: DocumentListLabels
|
||||
readonly index: IndexFormLabels
|
||||
readonly search: SearchFormLabels
|
||||
/** The project's automatic PDF indexing: the collection it fills, or null when it is off. */
|
||||
readonly autoIndex: { readonly current: string | null; readonly labels: AutoIndexFormLabels }
|
||||
}
|
||||
|
||||
export type KnowledgePage = ProjectPage<TransportCaller, I18n, Container, KnowledgePageData>
|
||||
|
||||
const indexFormLabels = (t: (key: string, values?: Record<string, string | number>) => string): IndexFormLabels => ({
|
||||
heading: t("upload.heading"),
|
||||
label: t("upload.label"),
|
||||
hint: t("upload.hint"),
|
||||
submit: t("upload.submit"),
|
||||
busy: t("upload.busy"),
|
||||
done: t("upload.done"),
|
||||
})
|
||||
|
||||
const documentStatusKey: Record<DocumentView["status"], string> = {
|
||||
queued: "documents.statusQueued",
|
||||
indexing: "documents.statusIndexing",
|
||||
indexed: "documents.statusIndexed",
|
||||
failed: "documents.statusFailed",
|
||||
cancelled: "documents.statusCancelled",
|
||||
}
|
||||
|
||||
export const knowledgePage: KnowledgePage = projectPage({
|
||||
async load({ slugs, segments, caller, i18n }): Promise<KnowledgePageData> {
|
||||
const t = (key: string, values?: Record<string, string | number>) => i18n.message(`knowledge.${key}`, values)
|
||||
if (segments.length > 0) throw notFound("no such page")
|
||||
|
||||
const [{ collections }, status, auto] = await Promise.all([
|
||||
caller.call(collectionList, slugs),
|
||||
caller.call(knowledgeStatus, slugs),
|
||||
caller.call(autoIndexGet, slugs),
|
||||
])
|
||||
const rows = await Promise.all(
|
||||
collections.map(async (collection: CollectionView): Promise<KnowledgeCollectionRow> => {
|
||||
const { documents } = await caller.call(documentList, { ...slugs, collectionName: collection.name })
|
||||
return { name: collection.name, documents }
|
||||
}),
|
||||
)
|
||||
|
||||
return {
|
||||
title: t("title"),
|
||||
intro: t("intro"),
|
||||
status: {
|
||||
enabled: status.enabled,
|
||||
labels: statusLabels(t, status),
|
||||
state: serviceStateLabel(t, status.service.state),
|
||||
reasons: status.service.reasons,
|
||||
...(status.service.refusal === undefined
|
||||
? {}
|
||||
: {
|
||||
refusal: {
|
||||
code: status.service.refusal,
|
||||
text: refusalLabels(t)[status.service.refusal],
|
||||
},
|
||||
}),
|
||||
},
|
||||
collections: rows,
|
||||
documents: {
|
||||
caption: t("documents.heading"),
|
||||
file: t("documents.file"),
|
||||
status: t("documents.status"),
|
||||
error: t("documents.error"),
|
||||
statuses: Object.fromEntries(
|
||||
Object.entries(documentStatusKey).map(([status, key]) => [status, t(key)]),
|
||||
) as DocumentListLabels["statuses"],
|
||||
failures: failureLabels(t),
|
||||
},
|
||||
index: indexFormLabels(t),
|
||||
search: searchFormLabels(t),
|
||||
autoIndex: {
|
||||
current: auto.collectionName,
|
||||
labels: {
|
||||
heading: t("autoIndex.heading"),
|
||||
hint: t("autoIndex.hint"),
|
||||
label: t("autoIndex.label"),
|
||||
off: t("autoIndex.off"),
|
||||
submit: t("autoIndex.submit"),
|
||||
busy: t("autoIndex.busy"),
|
||||
done: t("autoIndex.done"),
|
||||
},
|
||||
},
|
||||
}
|
||||
},
|
||||
|
||||
View({ slugs, locale, data }: ProjectPageViewProps<KnowledgePageData>): ReactElement {
|
||||
const { labels, enabled, state, reasons, refusal } = data.status
|
||||
return (
|
||||
<Page>
|
||||
<PageHeader title={data.title} description={data.intro} />
|
||||
<PageSection title={labels.heading} variant="card">
|
||||
{refusal ? (
|
||||
<Alert variant="warning" role="status" data-state={refusal.code}>
|
||||
<AlertDescription>{refusal.text}</AlertDescription>
|
||||
</Alert>
|
||||
) : null}
|
||||
{enabled ? (
|
||||
<FactList
|
||||
facts={[
|
||||
{ label: labels.state, value: state },
|
||||
...(reasons.length > 0
|
||||
? [
|
||||
{
|
||||
label: labels.reasons,
|
||||
value: (
|
||||
<ul>
|
||||
{reasons.map((reason) => (
|
||||
<li key={reason}>{reason}</li>
|
||||
))}
|
||||
</ul>
|
||||
),
|
||||
},
|
||||
]
|
||||
: []),
|
||||
{ label: labels.indexing, value: labels.documents },
|
||||
]}
|
||||
/>
|
||||
) : (
|
||||
<p role="status">{labels.flagOff}</p>
|
||||
)}
|
||||
</PageSection>
|
||||
{enabled && data.collections.length > 0 ? (
|
||||
<AutoIndexForm
|
||||
slugs={slugs}
|
||||
collections={data.collections.map((collection) => collection.name)}
|
||||
current={data.autoIndex.current}
|
||||
labels={data.autoIndex.labels}
|
||||
/>
|
||||
) : null}
|
||||
{enabled
|
||||
? data.collections.map((collection) => (
|
||||
<PageSection key={collection.name} title={collection.name}>
|
||||
<IndexForm slugs={slugs} collectionName={collection.name} labels={data.index} />
|
||||
<SearchForm slugs={slugs} collectionName={collection.name} labels={data.search} locale={locale} />
|
||||
<CollectionDocuments
|
||||
slugs={slugs}
|
||||
collectionName={collection.name}
|
||||
seed={collection.documents}
|
||||
labels={data.documents}
|
||||
/>
|
||||
</PageSection>
|
||||
))
|
||||
: null}
|
||||
</Page>
|
||||
)
|
||||
},
|
||||
})
|
||||
@@ -0,0 +1,91 @@
|
||||
"use client"
|
||||
|
||||
// The search form: one question, cited answers. The disclosure the EU AI Act Article 50 asks for
|
||||
// rides with the results, and every answer names its source and chunk — a person checks a passage
|
||||
// against the file, never a model's word.
|
||||
|
||||
import { formatMessage } from "@conusai-kit/i18n/format"
|
||||
import { AiDisclosureBadge, type ProjectSlugs } from "@conusai-kit/ui"
|
||||
import { useDataClient } from "@conusai-kit/ui/data"
|
||||
import { FormLayout, type FormValues, PageSection, TextField } from "@conusai-kit/ui/patterns"
|
||||
import { Alert, AlertDescription } from "@conusai-kit/ui/primitives"
|
||||
import { type ReactElement, useState } from "react"
|
||||
import type { SearchResult } from "../contract/index.js"
|
||||
import { type RefusalNotice, refusalOf, type SearchFormLabels } from "./labels.js"
|
||||
|
||||
export function SearchForm({
|
||||
slugs,
|
||||
collectionName,
|
||||
labels,
|
||||
locale,
|
||||
}: {
|
||||
slugs: ProjectSlugs
|
||||
collectionName: string
|
||||
labels: SearchFormLabels
|
||||
locale: string
|
||||
}): ReactElement {
|
||||
const client = useDataClient()
|
||||
const [results, setResults] = useState<SearchResult[] | null>(null)
|
||||
const [refused, setRefused] = useState<RefusalNotice | undefined>(undefined)
|
||||
|
||||
const search = async ({ query = "" }: FormValues): Promise<void> => {
|
||||
setResults(null)
|
||||
setRefused(undefined)
|
||||
try {
|
||||
const { value: answer } = await client.call<{ results: SearchResult[] }>("knowledge.search", {
|
||||
...slugs,
|
||||
collectionName,
|
||||
query,
|
||||
})
|
||||
setResults(answer.results)
|
||||
} catch (error) {
|
||||
// A refusal of the service has its own words. Anything else is FormLayout's to show.
|
||||
const notice = refusalOf(labels.refusals, error)
|
||||
if (notice === undefined) throw error
|
||||
setRefused(notice)
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<PageSection title={labels.heading} level={3} variant="card">
|
||||
<AiDisclosureBadge label={labels.disclosure} />
|
||||
<FormLayout submitLabel={labels.submit} busyLabel={labels.busy} onSubmit={search}>
|
||||
<TextField
|
||||
name="query"
|
||||
type="search"
|
||||
label={labels.label}
|
||||
placeholder={labels.placeholder}
|
||||
required
|
||||
maxLength={500}
|
||||
/>
|
||||
</FormLayout>
|
||||
{refused ? (
|
||||
<Alert variant="warning" data-state={refused.code}>
|
||||
<AlertDescription>{refused.text}</AlertDescription>
|
||||
</Alert>
|
||||
) : null}
|
||||
{results === null ? null : results.length === 0 ? (
|
||||
<p>{labels.empty}</p>
|
||||
) : (
|
||||
<ol aria-label={formatMessage(labels.results, { count: results.length }, locale)}>
|
||||
{results.map((result, at) => (
|
||||
<li key={`${result.fileId}-${result.chunk}-${at}`}>
|
||||
<blockquote
|
||||
data-knowledge-chunk={result.chunkId}
|
||||
data-source-page={result.page}
|
||||
data-source-block={result.blockId}
|
||||
>
|
||||
{result.snippet}
|
||||
</blockquote>
|
||||
<p>
|
||||
{labels.source}: {result.filename} · {labels.chunk.replace("{chunk}", String(result.chunk))}
|
||||
{result.page === undefined ? null : ` · ${labels.page.replace("{page}", String(result.page))}`}
|
||||
{result.blockId ? <code> {result.blockId}</code> : null}
|
||||
</p>
|
||||
</li>
|
||||
))}
|
||||
</ol>
|
||||
)}
|
||||
</PageSection>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
"use client"
|
||||
|
||||
import { Button } from "@conusai-kit/ui/primitives"
|
||||
|
||||
// The enablement controls of the organization's settings page: the current state, the enable
|
||||
// button with its typed refusal reasons (plan, policy, quota), and disable — which preserves
|
||||
// every collection, document and the service's index.
|
||||
|
||||
import { describeError, useWrite } from "@conusai-kit/ui/data"
|
||||
import { PageSection } from "@conusai-kit/ui/patterns"
|
||||
import { type ReactElement, useState } from "react"
|
||||
|
||||
export interface SettingsSectionLabels {
|
||||
readonly heading: string
|
||||
readonly state: string
|
||||
readonly stateOn: string
|
||||
readonly stateOff: string
|
||||
readonly backend: string
|
||||
readonly backendPlatform: string
|
||||
readonly enable: string
|
||||
readonly enabling: string
|
||||
readonly disable: string
|
||||
readonly disabling: string
|
||||
readonly enabledDone: string
|
||||
readonly disabledDone: string
|
||||
readonly planRequired: string
|
||||
readonly featureDisabled: string
|
||||
readonly quotaExhausted: string
|
||||
readonly upgrade: string
|
||||
readonly hint: string
|
||||
readonly failed: string
|
||||
}
|
||||
|
||||
type View = { kind: "idle" } | { kind: "busy" } | { kind: "message"; text: string; alert: boolean }
|
||||
|
||||
export function SettingsSection({
|
||||
org,
|
||||
initial,
|
||||
labels,
|
||||
}: {
|
||||
org: string
|
||||
initial: { enabled: boolean; backend: string | null }
|
||||
labels: SettingsSectionLabels
|
||||
}): ReactElement {
|
||||
const enableAi = useWrite<{ org: string }, unknown>("knowledge.settings.enableDocumentAi")
|
||||
const disableAi = useWrite<{ org: string }, unknown>("knowledge.settings.disableDocumentAi")
|
||||
const [enabled, setEnabled] = useState(initial.enabled)
|
||||
const [view, setView] = useState<View>({ kind: "idle" })
|
||||
|
||||
const enable = async (): Promise<void> => {
|
||||
setView({ kind: "busy" })
|
||||
try {
|
||||
await enableAi.mutate({ org })
|
||||
setEnabled(true)
|
||||
setView({ kind: "message", text: labels.enabledDone, alert: false })
|
||||
} catch (error) {
|
||||
const code = (error as { problem?: { code?: string } } | null)?.problem?.code ?? "unknown"
|
||||
if (code === "plan_required") {
|
||||
setView({ kind: "message", text: `${labels.planRequired} ${labels.upgrade}`, alert: true })
|
||||
} else if (code === "quota_exhausted") {
|
||||
setView({ kind: "message", text: labels.quotaExhausted, alert: true })
|
||||
} else if (code === "feature_disabled") {
|
||||
setView({ kind: "message", text: labels.featureDisabled, alert: true })
|
||||
} else {
|
||||
setView({ kind: "message", text: describeError(error), alert: true })
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const disable = async (): Promise<void> => {
|
||||
setView({ kind: "busy" })
|
||||
try {
|
||||
await disableAi.mutate({ org })
|
||||
setEnabled(false)
|
||||
setView({ kind: "message", text: labels.disabledDone, alert: false })
|
||||
} catch (error) {
|
||||
setView({ kind: "message", text: describeError(error), alert: true })
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<PageSection title={labels.heading}>
|
||||
<p>{labels.hint}</p>
|
||||
<dl>
|
||||
<dt>{labels.state}</dt>
|
||||
<dd>{enabled ? labels.stateOn : labels.stateOff}</dd>
|
||||
{initial.backend !== null ? (
|
||||
<>
|
||||
<dt>{labels.backend}</dt>
|
||||
<dd>{labels.backendPlatform}</dd>
|
||||
</>
|
||||
) : null}
|
||||
</dl>
|
||||
{view.kind === "message" ? <p role={view.alert ? "alert" : "status"}>{view.text}</p> : null}
|
||||
{enabled ? (
|
||||
<Button type="button" disabled={view.kind === "busy"} onClick={() => void disable()}>
|
||||
{view.kind === "busy" ? labels.disabling : labels.disable}
|
||||
</Button>
|
||||
) : (
|
||||
<Button type="button" disabled={view.kind === "busy"} onClick={() => void enable()}>
|
||||
{view.kind === "busy" ? labels.enabling : labels.enable}
|
||||
</Button>
|
||||
)}
|
||||
</PageSection>
|
||||
)
|
||||
}
|
||||
Reference in New Issue
Block a user