application tier

`application-wiki`

The wiki: external research — articles, YouTube, podcasts, books, web pages — fetched, DLP-screened, chunked and embedded into a searchable knowledge index (database: org_knowledge), plus the decision/session/lesson ledger. An application, not a tool: examples/staff-host already mounts its read model at /wiki/stats. NOT the source-code corpus (tools-corpus, database corpus_sourcecode).

The wiki: external research — articles, YouTube, podcasts, books, web pages — fetched, DLP-screened, chunked and embedded into a searchable knowledge index (database: org_knowledge), plus the decision/session/lesson ledger. An application, not a tool: examples/staff-host already mounts its read model at /wiki/stats. NOT the source-code corpus (tools-corpus, database corpus_sourcecode).

Tier	application
Role	application
Path	`crates/application/wiki`
Edition	2021
Targets	wiki, application_wiki, e2e_ingest_search, e2e_ingest_transcripts, hook_turn_live
Public items	102 across 12 modules
Tests	87

What it is for

tools-org-knowledge — library half of the org knowledge CLI (Sprint 3.4).

markdown: YAML front-matter extraction + ATX-heading pre-split with

FILE-ABSOLUTE char offsets, then per-section chunking via infrastructure_embeddings::chunk_text (2000/320 defaults).

embedder: the Gate 1.5 Q4 adapter wiring infrastructure_embeddings::Embedder

(dev DeterministicEmbedder; ONNX BGE-m3 behind the ort feature) into infrastructure_knowledge_index::KnowledgeEmbedder.

walk: containment-safe docs/**/*.md walker (canonicalized root,

symlink escapes refused + reported).

ingest: hash-skip / rename-detect / tombstone / embed-publish

pipeline over infrastructure_knowledge_index.

cli: the clap surface + --json envelope (schema_version: 1) +

boundary validation for JSON-shaped arguments.

federate: ledger rows → knowledge index (canonical plain-text

rendering, corpus ledger, source_ref='decision:<uuid>' etc.; superseding re-indexes both ends).

web: ingest-url host wiring over infrastructure_web_ingest — builds

this CLI's Fetcher (store dir, rate limiter, byte cap) and dispatches.

Capabilities

`BookIngestReport`

Book source connector (sprint 3.39, Phase 1): index the books we own

Item
`pub struct BookIngestReport`
`async fn run_ingest_books <E : KnowledgeEmbedder + ? Sized>(kn : & PgPool, inv : & PgPool, embedder : & E, tenant : Uuid, corpus : & str, full : bool,) -> Result <BookIngestReport, KnowledgeError>`

`BookRow`

Book source connector (sprint 3.39, Phase 1): index the books we own

Item
`pub struct BookRow`
`fn book_search_text(book : & BookRow) -> String`
`fn book_to_doc(book : & BookRow, tenant : Uuid, corpus : & str) ->(DocSource, Vec <ChunkInput>)`

`cli (other)`

CLI surface (T5): clap argument types + pure boundary helpers.

Item
`pub const SCHEMA_VERSION: u32`
`fn envelope(command : & str, data : serde_json::Value) -> serde_json::Value`
`fn parse_alternatives(raw : Option <& str>) -> Result <serde_json::Value, String>`
`fn parse_artifacts(raw : Option <& str>) -> Result <serde_json::Value, String>`

`Cli`

CLI surface (T5): clap argument types + pure boundary helpers.

Item
`pub struct Cli`

`Command`

CLI surface (T5): clap argument types + pure boundary helpers.

Item
`pub enum Command`

`DecisionAddArgs`

CLI surface (T5): clap argument types + pure boundary helpers.

Item
`pub struct DecisionAddArgs`

`DecisionCmd`

CLI surface (T5): clap argument types + pure boundary helpers.

Item
`pub enum DecisionCmd`

`DecisionsCmd`

CLI surface (T5): clap argument types + pure boundary helpers.

Item
`pub enum DecisionsCmd`

`LessonCmd`

CLI surface (T5): clap argument types + pure boundary helpers.

Item
`pub enum LessonCmd`

`SessionCmd`

CLI surface (T5): clap argument types + pure boundary helpers.

Item
`pub enum SessionCmd`

`federate (other)`

Federation (T6): ledger rows → knowledge index.

Item
`pub const LEDGER_CORPUS: & str`
`fn decision_source_ref(id : Uuid) -> String`
`fn session_source_ref(id : Uuid) -> String`
`fn lesson_source_ref(id : Uuid) -> String`
`fn render_decision(d : & Decision) -> String`
`fn render_session(s : & SessionRecord) -> String`
`fn render_lesson(l : & Lesson) -> String`
`fn decision_front_matter(d : & Decision) -> serde_json::Value`
`async fn run_federation <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, tenant : Uuid,) -> Result <FederationReport, FederateError>`

`FederateError`

Federation (T6): ledger rows → knowledge index.

Item
`pub enum FederateError`
`async fn federate_decision <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, d : & Decision,) -> Result <IndexOutcome, FederateError>`
`async fn federate_session <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, s : & SessionRecord,) -> Result <IndexOutcome, FederateError>`
`async fn federate_lesson <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, l : & Lesson,) -> Result <IndexOutcome, FederateError>`
`async fn federate_supersede <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, tenant : Uuid, old_id : Uuid, new_id : Uuid,) -> Result <(), FederateError>`

`FederationReport`

Federation (T6): ledger rows → knowledge index.

Item
`pub struct FederationReport`

`ingest (other)`

Ingest pipeline (memo steps 1-9): walk → hash → diff against the index →

Item
`async fn run_ingest <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, tenant : Uuid, repo_root : & Path, corpus : & str, full : bool,) -> Result <IngestReport, IngestError>`

`IngestError`

Ingest pipeline (memo steps 1-9): walk → hash → diff against the index →

Item
`pub enum IngestError`

`IngestReport`

Ingest pipeline (memo steps 1-9): walk → hash → diff against the index →

Item
`pub struct IngestReport`

`ingest_transcripts (other)`

Transcript ingest + survey orchestration (Sprint 3.15, T1/T4).

Item
`fn project_is_excluded(project_dir : & str, excludes : & String) -> bool`
`fn walk_sessions(root : & Path, excludes : & String,) -> Result <Vec <SessionFile>, TranscriptIngestError>`
`fn session_content_hash(turns : & SelectedTurn) -> String`
`fn turns_to_chunks(turns : & SelectedTurn) -> Vec <ChunkInput>`
`async fn run_survey(root : & Path, caps : & TranscriptCaps, excludes : & String,) -> Result <TranscriptReport, TranscriptIngestError>`
`async fn run_ingest_transcripts <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, tenant : Uuid, root : & Path, corpus : & str, caps : & TranscriptCaps, full : bool, excludes : & String,) -> Result <TranscriptReport, TranscriptIngestError>`
`fn model_backfill_plan(turns : & SelectedTurn) -> Vec <(String, String, Option <TurnUsage>)>`
`async fn run_backfill_models(corpus : & PgPool, root : & Path, caps : & TranscriptCaps, excludes : & String,) -> Result <BackfillReport, TranscriptIngestError>`
`async fn run_hook_turn(corpus : & PgPool, transcript_path : & Path, sessions_root : & Path, caps : & TranscriptCaps,) -> Result <HookOutcome, TranscriptIngestError>`

`BackfillReport`

Transcript ingest + survey orchestration (Sprint 3.15, T1/T4).

Item
`pub struct BackfillReport`

`HookOutcome`

Transcript ingest + survey orchestration (Sprint 3.15, T1/T4).

Item
`pub enum HookOutcome`

`SessionFile`

Transcript ingest + survey orchestration (Sprint 3.15, T1/T4).

Item
`pub struct SessionFile`

`TranscriptCaps`

Transcript ingest + survey orchestration (Sprint 3.15, T1/T4).

Item
`pub struct TranscriptCaps`
`TranscriptCaps :: fn default() -> Self`

`TranscriptIngestError`

Transcript ingest + survey orchestration (Sprint 3.15, T1/T4).

Item
`pub enum TranscriptIngestError`

`TranscriptReport`

Transcript ingest + survey orchestration (Sprint 3.15, T1/T4).

Item
`pub struct TranscriptReport`
`fn report_to_json(report : & TranscriptReport) -> serde_json::Value`

`research (other)`

Research-corpus ingest (sprint 3.37, AC-1) + the deterministic

Item
`pub const RESEARCH_CORPUS: & str`
`pub const SOURCE_KIND_RESEARCH: & str`
`fn parse_ledger(text : & str) ->(Vec <LedgerRow>, Vec <LedgerViolation>)`
`async fn run_ingest_research <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, tenant : Uuid, root : & Path, corpus : & str, full : bool,) -> Result <ResearchReport, ResearchIngestError>`

`LedgerRow`

Research-corpus ingest (sprint 3.37, AC-1) + the deterministic

Item
`pub struct LedgerRow`
`fn research_delta(corpus_ids : & String, rows : & LedgerRow) ->(Vec <String>, Vec <String>)`

`LedgerViolation`

Research-corpus ingest (sprint 3.37, AC-1) + the deterministic

Item
`pub struct LedgerViolation`

`ResearchHeader`

Research-corpus ingest (sprint 3.37, AC-1) + the deterministic

Item
`pub struct ResearchHeader`
`fn is_headerless(h : & ResearchHeader) -> bool`
`fn parse_research_header(text : & str) -> ResearchHeader`

`ResearchIngestError`

Research-corpus ingest (sprint 3.37, AC-1) + the deterministic

Item
`pub enum ResearchIngestError`

`ResearchReport`

Research-corpus ingest (sprint 3.37, AC-1) + the deterministic

Item
`pub struct ResearchReport`

`scope (other)`

What this tool is for, stated by the tool itself (sprint 4.21).

Item
`pub const PURPOSE: & str`
`pub const NOT_FOR: & str`
`pub const WIKI_COMMANDS: & & str`
`fn is_wiki_command(command : & str) -> bool`
`pub const MISPLACED_COMMANDS: & (& str, & str)`
`fn misplaced_owner(command : & str) -> Option <& 'static str>`
`fn banner(command : Option <& str>) -> String`

`transcripts (other)`

Session-transcript selection + redaction (Sprint 3.15, T2/T3).

Item
`pub const TRANSCRIPT_CORPUS: & str`
`pub const SOURCE_KIND_TRANSCRIPT: & str`
`pub const DEFAULT_TURN_BYTE_CAP: usize`
`pub const TRUNCATION_MARKER: & str`
`pub const CHUNK_KIND_TURN: & str`
`fn select_turn(line : & Value, line_index : usize, cap : usize, counts : & mut SelectionCounts,) -> Option <SelectedTurn>`
`pub const HOOK_TAIL_BYTES: u64`
`fn source_ref_of(transcript_path : & Path, sessions_root : & Path) -> String`
`async fn read_transcript_tail(path : & Path, max_bytes : u64) -> Result <String, std::io::Error>`
`fn redact_text(text : & str) ->(String, usize)`
`fn turn_hash(role : & str, content : & str) -> String`

`ExcludedClass`

Session-transcript selection + redaction (Sprint 3.15, T2/T3).

Item
`pub enum ExcludedClass`
`ExcludedClass :: fn as_str(self) -> & 'static str`

`SelectedTurn`

Session-transcript selection + redaction (Sprint 3.15, T2/T3).

Item
`pub struct SelectedTurn`
`fn last_assistant_turn(lines : & str, cap : usize) -> Option <(SelectedTurn, Option <uuid::Uuid>)>`

`SelectionCounts`

Session-transcript selection + redaction (Sprint 3.15, T2/T3).

Item
`pub struct SelectionCounts`

`SessionRead`

Session-transcript selection + redaction (Sprint 3.15, T2/T3).

Item
`pub struct SessionRead`
`async fn read_session(path : & Path, turn_byte_cap : usize, max_line_bytes : usize, read_timeout : Duration, deadline : Option <std::time::Instant>,) -> Result <SessionRead, std::io::Error>`

`TurnUsage`

Session-transcript selection + redaction (Sprint 3.15, T2/T3).

Item
`pub struct TurnUsage`
`TurnUsage :: fn from_line(line : & Value) -> Option <Self>`

`walk (other)`

Containment-safe docs/**/*.md walker (memo step 1).

Item
`fn walk_markdown_docs(repo_root : & Path) -> Result <WalkOutcome, WalkError>`

`SkippedEntry`

Containment-safe docs/**/*.md walker (memo step 1).

Item
`pub struct SkippedEntry`

`WalkError`

Containment-safe docs/**/*.md walker (memo step 1).

Item
`pub enum WalkError`

`WalkOutcome`

Containment-safe docs/**/*.md walker (memo step 1).

Item
`pub struct WalkOutcome`

`WalkedFile`

Containment-safe docs/**/*.md walker (memo step 1).

Item
`pub struct WalkedFile`

`web (other)`

ingest-url host wiring (T3): builds the Fetcher this CLI process uses

Item
`fn default_store_dir() -> PathBuf`

`WebCliError`

ingest-url host wiring (T3): builds the Fetcher this CLI process uses

Item
`pub enum WebCliError`
`fn build_web_fetcher(store_dir : & Path,) -> Result <(Fetcher, Arc <dyn StorageBackend>), WebCliError>`
`async fn run_ingest_url <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, tenant : Uuid, corpus : & str, store_dir : & Path, url : & Url, force : bool,) -> Result <WebIngestReport, WebCliError>`

`wiki_stats (other)`

Wiki-dashboard reporting over knowledge_doc (sprint 3.101 rescope,

Item
`async fn last_ingested_at(pool : & PgPool) -> Result <Option <DateTime <Utc>>, sqlx::Error>`
`async fn distinct_source_links(pool : & PgPool) -> Result <Vec <String>, sqlx::Error>`

`youtube_stats (other)`

YouTube-specific reporting over the generic research corpus.

Item
`async fn youtube_video_count(pool : & PgPool) -> Result <i64, sqlx::Error>`

How to use it

No examples/ target and no doctest in this crate's rustdoc. The tests listed under Verification are the closest executable usage.

Module structure

application_wiki

book_ingest
cli
federate
ingest
ingest_transcripts
research
scope
transcripts
walk
web
wiki_stats
youtube_stats

flowchart TD
  n_application_wiki["application_wiki"]
  n_application_wiki --> n_book_ingest["book_ingest"]
  n_application_wiki --> n_cli["cli"]
  n_application_wiki --> n_federate["federate"]
  n_application_wiki --> n_ingest["ingest"]
  n_application_wiki --> n_ingest_transcripts["ingest_transcripts"]
  n_application_wiki --> n_research["research"]
  n_application_wiki --> n_scope["scope"]
  n_application_wiki --> n_transcripts["transcripts"]
  n_application_wiki --> n_walk["walk"]
  n_application_wiki --> n_web["web"]
  n_application_wiki --> n_wiki_stats["wiki_stats"]
  n_application_wiki --> n_youtube_stats["youtube_stats"]

Public surface

`book_ingest`

Item	What it is
`pub struct BookRow`	One book row read from `inventory_ai` (the fields we index; the rest of the catalog row stays in its own DB and is dereferenced on demand).
`fn book_search_text(book : & BookRow) -> String`	The searchable text for a book: the fields a subject/title/author query should match, one per line, empty fields skipped
`fn book_to_doc(book : & BookRow, tenant : Uuid, corpus : & str) ->(DocSource, Vec <ChunkInput>)`	Pure transform: an `inventory_ai` book row → the `DocSource` + one `ChunkInput` the knowledge index consumes
`pub struct BookIngestReport`	What a book-ingest pass did.
`async fn run_ingest_books <E : KnowledgeEmbedder + ? Sized>(kn : & PgPool, inv : & PgPool, embedder : & E, tenant : Uuid, corpus : & str, full : bool,) -> Result <BookIngestReport, KnowledgeError>`	Read the books we own from `inventory_ai` (SELECT-only) and index each into the knowledge index, mirroring `run_ingest`'s per-doc shape: hash-skip + mixed-model refusal live in `IndexService`, so a re-run embeds only changed rows

`cli`

Item	What it is
`pub const SCHEMA_VERSION: u32`	JSON envelope schema version emitted by every `--json` output.
`pub struct Cli`	Org knowledge index CLI (Sprint 3.4).
`pub enum Command`	Top-level subcommands.
`pub enum DecisionsCmd`	`decisions ...` (reads).
`pub enum DecisionCmd`	`decision ...` (writes).
`pub struct DecisionAddArgs`	Arguments for `decision add`.
`pub enum SessionCmd`	`session ...`.
`pub enum LessonCmd`	`lesson ...`.
`fn envelope(command : & str, data : serde_json::Value) -> serde_json::Value`	Wrap subcommand output in the versioned JSON envelope.
`fn parse_alternatives(raw : Option <& str>) -> Result <serde_json::Value, String>`	Parse + validate `--alternatives-json` at the boundary: `None` → ; `Some` must parse as JSON AND satisfy the ledger's `{"option","reason"}` shape
`fn parse_artifacts(raw : Option <& str>) -> Result <serde_json::Value, String>`	Parse `--artifacts-json`: `None` → ; `Some` must be a JSON array

`federate`

Item	What it is
`pub const LEDGER_CORPUS: & str`	The corpus every federated ledger row lands in.
`pub enum FederateError`	Failures of a federation pass.
`fn decision_source_ref(id : Uuid) -> String`	`source_ref` for a federated decision row.
`fn session_source_ref(id : Uuid) -> String`	`source_ref` for a federated session row.
`fn lesson_source_ref(id : Uuid) -> String`	`source_ref` for a federated lesson row.
`fn render_decision(d : & Decision) -> String`	Render a decision to its canonical plain-text representation
`fn render_session(s : & SessionRecord) -> String`	Render a session to its canonical plain-text representation.
`fn render_lesson(l : & Lesson) -> String`	Render a lesson to its canonical plain-text representation.
`fn decision_front_matter(d : & Decision) -> serde_json::Value`	Front matter for a federated decision doc
`async fn federate_decision <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, d : & Decision,) -> Result <IndexOutcome, FederateError>`	Upsert one decision row into the knowledge index
`async fn federate_session <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, s : & SessionRecord,) -> Result <IndexOutcome, FederateError>`	Upsert one session row into the knowledge index
`async fn federate_lesson <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, l : & Lesson,) -> Result <IndexOutcome, FederateError>`	Upsert one lesson row into the knowledge index
`async fn federate_supersede <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, tenant : Uuid, old_id : Uuid, new_id : Uuid,) -> Result <(), FederateError>`	Re-index BOTH ends of a supersession (memo T6): the old row's rendered text + front matter now say `superseded`, so it drops out of default search; the new row indexes as `active`
`pub struct FederationReport`	What one full federation pass did (serializable for `--json`).
`async fn run_federation <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, tenant : Uuid,) -> Result <FederationReport, FederateError>`	Full federation pass: render + upsert EVERY ledger row for `tenant`

`ingest`

Item	What it is
`pub enum IngestError`	Ingest-level failures that abort the whole run (per-file problems are reported in `IngestReport::errors` instead).
`pub struct IngestReport`	What one ingest run did (serializable for `--json`).
`async fn run_ingest <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, tenant : Uuid, repo_root : & Path, corpus : & str, full : bool,) -> Result <IngestReport, IngestError>`	Run one ingest pass over `<repo_root>/docs/*/.md` into `corpus`

`ingest_transcripts`

Item	What it is
`pub struct TranscriptCaps`	Caps applied to every session file read (Gate 1.5.6 input limits).
`TranscriptCaps :: fn default() -> Self`	—
`pub struct SessionFile`	One discovered session file.
`pub enum TranscriptIngestError`	Errors that abort a transcript run (fail loud, never commit partial).
`pub struct TranscriptReport`	Aggregate report for a survey or ingest run
`fn project_is_excluded(project_dir : & str, excludes : & String) -> bool`	True when a project directory name matches any exclude pattern (case-insensitive substring — operator decision 2026-08-08: case-file projects stay out of the corpus, including future ones).
`fn walk_sessions(root : & Path, excludes : & String,) -> Result <Vec <SessionFile>, TranscriptIngestError>`	Discover session files under `root`, RECURSIVELY (operator scope amendment 2026-08-08: subagent/workflow transcripts live at `<project>/<session-uuid>/subagents/*/.jsonl`)
`fn session_content_hash(turns : & SelectedTurn) -> String`	A session's doc-level content hash: sha256 over its ordered turn hashes
`fn turns_to_chunks(turns : & SelectedTurn) -> Vec <ChunkInput>`	Map selected turns to index chunks: chunk_index = turn_index (physical line), chunk_kind = 'turn', per-chunk hash, provenance meta.
`async fn run_survey(root : & Path, caps : & TranscriptCaps, excludes : & String,) -> Result <TranscriptReport, TranscriptIngestError>`	Survey (T1): walk + read + select, NO DB and NO embedding
`async fn run_ingest_transcripts <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, tenant : Uuid, root : & Path, corpus : & str, caps : & TranscriptCaps, full : bool, excludes : & String,) -> Result <TranscriptReport, TranscriptIngestError>`	Ingest (T4): survey's walk + read, then publish each session's turns via the incremental index path
`fn model_backfill_plan(turns : & SelectedTurn) -> Vec <(String, String, Option <TurnUsage>)>`	Render a report as the JSON payload for the CLI envelope
`pub struct BackfillReport`	What the backfill did, in `corpus_sourcecode`.
`async fn run_backfill_models(corpus : & PgPool, root : & Path, caps : & TranscriptCaps, excludes : & String,) -> Result <BackfillReport, TranscriptIngestError>`	Story 4.74, option 3, as the operator ruled it: the model goes into `corpus_sourcecode.assistant_turn`, not into the wiki
`pub enum HookOutcome`	What the Stop hook did with one turn (story 4.74, work item 5).
`async fn run_hook_turn(corpus : & PgPool, transcript_path : & Path, sessions_root : & Path, caps : & TranscriptCaps,) -> Result <HookOutcome, TranscriptIngestError>`	Stage the last assistant turn of a transcript into `corpus_sourcecode` and read it at once
`fn report_to_json(report : & TranscriptReport) -> serde_json::Value`	—

`research`

Item	What it is
`pub const RESEARCH_CORPUS: & str`	Corpus name for external research ideas.
`pub const SOURCE_KIND_RESEARCH: & str`	`source_kind` stamped on research docs.
`pub struct ResearchHeader`	Provenance parsed from an `ideas.md` plain-text header.
`fn is_headerless(h : & ResearchHeader) -> bool`	A header is absent only when neither provenance field is present
`fn parse_research_header(text : & str) -> ResearchHeader`	Parse the `Key: value` header lines (first blank line or heading ends the header)
`pub struct LedgerRow`	One ledger row from `docs/reference/RESEARCH-LEDGER.md`.
`pub struct LedgerViolation`	A ledger row that violates the discipline.
`fn parse_ledger(text : & str) ->(Vec <LedgerRow>, Vec <LedgerViolation>)`	Parse ledger rows: `\
`fn research_delta(corpus_ids : & String, rows : & LedgerRow) ->(Vec <String>, Vec <String>)`	The deterministic research-delta: corpus doc ids with no ledger row or a `pending` row, plus ledger rows citing docs that do not exist (rot).
`pub struct ResearchReport`	Report from a research ingest run.
`pub enum ResearchIngestError`	Errors that abort a research ingest.
`async fn run_ingest_research <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, tenant : Uuid, root : & Path, corpus : & str, full : bool,) -> Result <ResearchReport, ResearchIngestError>`	Ingest the research tree: one doc per markdown file, whole-file hash-skip, header provenance in `front_matter`, markdown chunking.

`scope`

Item	What it is
`pub const PURPOSE: & str`	What the tool is for
`pub const NOT_FOR: & str`	What the tool is NOT for, with the owner of each concern.
`pub const WIKI_COMMANDS: & & str`	Commands that are genuinely wiki work
`fn is_wiki_command(command : & str) -> bool`	Whether `command` is wiki work
`pub const MISPLACED_COMMANDS: & (& str, & str)`	Commands in this binary that are NOT wiki work, and where each belongs
`fn misplaced_owner(command : & str) -> Option <& 'static str>`	The owner of a misplaced command, if it is one.
`fn banner(command : Option <& str>) -> String`	The banner, printed to STDERR so machine-readable stdout is untouched.

`transcripts`

Item	What it is
`pub const TRANSCRIPT_CORPUS: & str`	Corpus name for session transcripts in the knowledge index.
`pub const SOURCE_KIND_TRANSCRIPT: & str`	`source_kind` stamped on transcript docs/chunks.
`pub const DEFAULT_TURN_BYTE_CAP: usize`	Default per-turn byte cap (sprint AC; configurable, never silent).
`pub const TRUNCATION_MARKER: & str`	Marker appended to any turn truncated by the byte cap.
`pub const CHUNK_KIND_TURN: & str`	Chunk kind stamped on selected turns (knowledge-index `chunk_kind`).
`pub enum ExcludedClass`	Why a line (or block) was excluded from the index.
`ExcludedClass :: fn as_str(self) -> & 'static str`	Stable lowercase name for reports and JSON output.
`pub struct SelectedTurn`	One turn selected for indexing.
`pub struct TurnUsage`	The token counts an assistant line reports for itself.
`TurnUsage :: fn from_line(line : & Value) -> Option <Self>`	Read `message.usage` from a line
`pub struct SelectionCounts`	Per-run selection bookkeeping
`fn select_turn(line : & Value, line_index : usize, cap : usize, counts : & mut SelectionCounts,) -> Option <SelectedTurn>`	Classify + select one parsed line
`pub const HOOK_TAIL_BYTES: u64`	The tail of a transcript the live hook reads: enough for the last turn of any session, and a hard cap on what one hook invocation will load.
`fn last_assistant_turn(lines : & str, cap : usize) -> Option <(SelectedTurn, Option <uuid::Uuid>)>`	Story 4.74, work item 5
`fn source_ref_of(transcript_path : & Path, sessions_root : & Path) -> String`	`<project>/<session-stem>` from a transcript path — the same shape the batch ingest gives a session, so hook rows and batch rows name a session the same way
`async fn read_transcript_tail(path : & Path, max_bytes : u64) -> Result <String, std::io::Error>`	Read the last `max_bytes` of a transcript, whole lines only: the bytes before the first newline in the window are a partial line and dropped
`fn redact_text(text : & str) ->(String, usize)`	Redact sensitive values via `infrastructure-dlp-detect` (default policy)
`fn turn_hash(role : & str, content : & str) -> String`	Hex sha256 over the stored form of a turn (`role` NUL `content`).
`pub struct SessionRead`	Outcome of reading one session file: selected turns + bookkeeping.
`async fn read_session(path : & Path, turn_byte_cap : usize, max_line_bytes : usize, read_timeout : Duration, deadline : Option <std::time::Instant>,) -> Result <SessionRead, std::io::Error>`	Read a session `.jsonl` through the bounded reader: per-line byte cap, read timeout, malformed lines (including a truncated final line) skipped and counted, never fatal to the file

`walk`

Item	What it is
`pub enum WalkError`	Walk-level failures (the walk itself could not run at all).
`pub struct WalkedFile`	One markdown file found by the walk.
`pub struct SkippedEntry`	A path the walk refused or could not process, with the reason (reported, never silently dropped).
`pub struct WalkOutcome`	Walk result: included markdown files (sorted by `rel_ref`) + skip reports.
`fn walk_markdown_docs(repo_root : & Path) -> Result <WalkOutcome, WalkError>`	Walk `docs/*/.md` under `repo_root` (canonicalized), refusing symlink escapes

`web`

Item	What it is
`pub enum WebCliError`	Failures specific to the CLI's `ingest-url` wiring (as opposed to the pipeline itself, `WebIngestError`).
`fn default_store_dir() -> PathBuf`	Default artifact store directory: the XDG cache dir's `tools-org-knowledge/web-store` (falls back to `.cache/tools-org-knowledge/web-store` under the current directory if no cache dir can be resolved -- e.g
`fn build_web_fetcher(store_dir : & Path,) -> Result <(Fetcher, Arc <dyn StorageBackend>), WebCliError>`	Build the `Fetcher` (+ its storage handle) this CLI uses for `ingest-url`: a `ReqwestExecutor` capped at `DEFAULT_BYTE_CAP` (8 MiB -- page- appropriate, per the story), `LocalStorage` rooted at the canonicalized `store_dir`, and a modest `InMemoryRateLimiter`
`async fn run_ingest_url <E : KnowledgeEmbedder + ? Sized>(pool : & PgPool, embedder : & E, tenant : Uuid, corpus : & str, store_dir : & Path, url : & Url, force : bool,) -> Result <WebIngestReport, WebCliError>`	Fetch `url` and ingest it into `corpus`, using a `Fetcher` built over `store_dir`

`wiki_stats`

Item	What it is
`async fn last_ingested_at(pool : & PgPool) -> Result <Option <DateTime <Utc>>, sqlx::Error>`	Timestamp of the most recently ingested `knowledge_doc` row, or `None` if the table is empty
`async fn distinct_source_links(pool : & PgPool) -> Result <Vec <String>, sqlx::Error>`	Distinct, non-null `source_url` values across every ingested doc, alphabetical

`youtube_stats`

Item	What it is
`async fn youtube_video_count(pool : & PgPool) -> Result <i64, sqlx::Error>`	Count of ingested `source_kind = 'research'` docs whose `front_matter->>'source_url'` names a YouTube URL (Rule 14 -- "AI never counts what code can count," direct answer to "how many YouTube videos have been ingested," 2026-08-13)

Re-exports. Exported here, defined elsewhere.

Export	Defined in
`{build_web_fetcher,default_store_dir,run_ingest_url,WebCliError}`	`web::{build_web_fetcher,default_store_dir,run_ingest_url,WebCliError}`
`{chunk_document,parse_and_chunk,parse_document,DocChunk,MarkdownError,ParsedDoc,Section,}`	`markdown::{chunk_document,parse_and_chunk,parse_document,DocChunk,MarkdownError,ParsedDoc,Section,}`
`{classify_source_kind,derive_title,derive_title_or_file_name,git_provenance,plan_walk_actions,GitProvenance,WalkActions,}`	`infrastructure_doc_ingest::walk_plan::{classify_source_kind,derive_title,derive_title_or_file_name,git_provenance,plan_walk_actions,GitProvenance,WalkActions,}`
`{classify_source_kind,derive_title,git_provenance,plan_walk_actions,run_ingest,GitProvenance,IngestError,IngestReport,WalkActions,}`	`ingest::{classify_source_kind,derive_title,git_provenance,plan_walk_actions,run_ingest,GitProvenance,IngestError,IngestReport,WalkActions,}`
`{db_stats,embedder,markdown}`	`infrastructure_doc_ingest::{db_stats,embedder,markdown}`
`{decision_front_matter,decision_source_ref,federate_decision,federate_lesson,federate_session,federate_supersede,lesson_source_ref,render_decision,render_lesson,render_session,run_federation,session_source_ref,FederateError,FederationReport,LEDGER_CORPUS,}`	`federate::{decision_front_matter,decision_source_ref,federate_decision,federate_lesson,federate_session,federate_supersede,lesson_source_ref,render_decision,render_lesson,render_session,run_federation,session_source_ref,FederateError,FederationReport,LEDGER_CORPUS,}`
`{default_embedder,KnowledgeEmbedderAdapter}`	`embedder::{default_embedder,KnowledgeEmbedderAdapter}`
`{envelope,parse_alternatives,parse_artifacts,Cli,SCHEMA_VERSION}`	`cli::{envelope,parse_alternatives,parse_artifacts,Cli,SCHEMA_VERSION}`
`{walk_markdown_docs,SkippedEntry,WalkError,WalkOutcome,WalkedFile}`	`walk::{walk_markdown_docs,SkippedEntry,WalkError,WalkOutcome,WalkedFile}`

Boundary

Reaches into foundation, infrastructure, operations.

Shares tier application with 120 other crates: application-agreements, application-ai, application-analytics, application-approvals, application-assessments, application-audit-log, application-auth, application-billing, … (120 total).

_What this crate deliberately does NOT own is a judgment. No committed registry records one for it, so none is stated here._

Where it sits

Tier (ontology)	`application`
Architectural role (taxonomy)	`application`
Location	`crates/application/wiki`
Vocabulary in force (lexicon)	`current`

Tier flow. Which tiers this crate's own edges cross.

flowchart LR
  n_application["application"] --> n_foundation["foundation"]
  n_application["application"] --> n_infrastructure["infrastructure"]
  n_application["application"] --> n_operations["operations"]

Dependencies

Runtime, in this workspace.

Crate	Tier	Optional	Only on
`application-core`	application	no	always
`foundation-audit-log`	foundation	no	always
`foundation-bounded-io`	foundation	no	always
`foundation-fs-metadata`	foundation	no	always
`infrastructure-dlp-detect`	infrastructure	no	always
`infrastructure-doc-ingest`	infrastructure	no	always
`infrastructure-embeddings`	infrastructure	no	always
`infrastructure-fetcher`	infrastructure	no	always
`infrastructure-knowledge-index`	infrastructure	no	always
`infrastructure-storage`	infrastructure	no	always
`infrastructure-web-ingest`	infrastructure	no	always
`operations-decision-ledger`	operations	no	always
`operations-rate-limit`	operations	no	always

Runtime, from outside the workspace.

Crate	Requirement	Features	Optional	Only on
`async-trait`	`^0.1`	—	no	always
`chrono`	`^0.4`	`serde`	no	always
`clap`	`^4`	`derive`	no	always
`dirs`	`^5`	—	no	always
`serde`	`^1`	`derive`, `derive`	no	always
`serde_json`	`^1`	—	no	always
`serde_yaml`	`^0.9`	—	no	always
`sha2`	`^0.10`	—	no	always
`sqlx`	`^0.8`	`runtime-tokio`, `postgres`, `chrono`, `uuid`, `json`	no	always
`thiserror`	`^2`	—	no	always
`tokio`	`^1`	`full`, `macros`, `rt-multi-thread`, `fs`	no	always
`url`	`^2`	—	no	always
`uuid`	`^1`	`v4`, `v7`, `serde`, `js`, `v4`, `serde`	no	always

Development, in this workspace.

Crate	Tier	Optional	Only on
`infrastructure-fetcher`	infrastructure	no	always
`tools-cli-conformance`	tools	no	always

Development, from outside the workspace.

Crate	Requirement	Features	Optional	Only on
`wiremock`	`^0.6`	—	no	always

Build. None.

Depended on by. Nothing in this workspace.

Signal flow — what reaches this crate, and what it reaches.

flowchart LR
  SELF["application-wiki"]
  SELF -->|development| n_infrastructure_fetcher["infrastructure-fetcher"]
  SELF -->|development| n_tools_cli_conformance["tools-cli-conformance"]
  SELF -->|runtime| n_application_core["application-core"]
  SELF -->|runtime| n_foundation_audit_log["foundation-audit-log"]
  SELF -->|runtime| n_foundation_bounded_io["foundation-bounded-io"]
  SELF -->|runtime| n_foundation_fs_metadata["foundation-fs-metadata"]
  SELF -->|runtime| n_infrastructure_dlp_detect["infrastructure-dlp-detect"]
  SELF -->|runtime| n_infrastructure_doc_ingest["infrastructure-doc-ingest"]
  SELF -->|runtime| n_infrastructure_embeddings["infrastructure-embeddings"]
  SELF -->|runtime| n_infrastructure_fetcher["infrastructure-fetcher"]
  SELF -->|runtime| n_infrastructure_knowledge_index["infrastructure-knowledge-index"]
  SELF -->|runtime| n_infrastructure_storage["infrastructure-storage"]
  SELF -->|runtime| n_infrastructure_web_ingest["infrastructure-web-ingest"]
  SELF -->|runtime| n_operations_decision_ledger["operations-decision-ledger"]
  SELF -->|runtime| n_operations_rate_limit["operations-rate-limit"]
  classDef self fill:#1f883d,stroke:#1f883d,color:#fff;
  class SELF self;

Feature flags

Feature	Enables	On by default
`ort`	`infrastructure-embeddings/ort`, `infrastructure-doc-ingest/ort`	no

flowchart LR
  n_ort["ort"] --> n_infrastructure_embeddings_ort["infrastructure-embeddings/ort"]
  n_ort["ort"] --> n_infrastructure_doc_ingest_ort["infrastructure-doc-ingest/ort"]

Targets

Kind	Name	Source
bin	`wiki`	`src/main.rs`
lib	`application_wiki`	`src/lib.rs`
test	`e2e_ingest_search`	`tests/e2e_ingest_search.rs`
test	`e2e_ingest_transcripts`	`tests/e2e_ingest_transcripts.rs`
test	`hook_turn_live`	`tests/hook_turn_live.rs`

Error model

Error type	Named by
`FederateError`	`federate_decision`, `federate_lesson`, `federate_session`, `federate_supersede`, `run_federation`
`IngestError`	`ResearchIngestError`, `TranscriptIngestError`, `run_backfill_models`, `run_hook_turn`, `run_ingest`, `run_ingest_research`, … (9 total)
`ResearchIngestError`	`run_ingest_research`
`TranscriptIngestError`	`run_backfill_models`, `run_hook_turn`, `run_ingest_transcripts`, `run_survey`, `walk_sessions`
`WalkError`	`walk_markdown_docs`
`WebCliError`	`build_web_fetcher`, `run_ingest_url`

Operational characteristics

Property	Evidence
async public surface	yes
async runtime	yes
database access	yes
network I/O	none detected
unsafe code	none detected
environment variables	yes

No unsafe block, unsafe fn, unsafe impl or unsafe trait was found by the parser anywhere in this crate's source.

Configuration

Variable	Read in
`CORPUS_SOURCECODE_APP_DB`	`src/main.rs`
`DATABASE_URL`	`src/main.rs`
`INVENTORY_DB`	`src/main.rs`
`KNOWLEDGE_TEST_DB`	`src/federate.rs`
`ORG_KNOWLEDGE_DB`	`src/main.rs`
`TEST_DATABASE_URL`	`src/wiki_stats.rs`

No workspace crate depends on this one.

Verification

Kind	Count
Unit tests	75
Integration tests	12
Examples	0
Doctests	0

Evidence by module. How often each public module is named by something executable.

Module	Tests	Examples	Consumers
`book_ingest`	5	0	0
`cli`	11	0	0
`federate`	15	0	0
`ingest`	3	0	0
`ingest_transcripts`	16	0	0
`research`	12	0	0
`scope`	7	0	0
`transcripts`	18	0	0
`walk`	5	0	0
`web`	4	0	0
`wiki_stats`	2	0	0
`youtube_stats`	1	0	0

What the tests establish, by name:

cross_repo_ingest_does_not_tombstone_other_repos_docs — tests/e2e_ingest_search.rs
ingest_twice_touch_one_search_with_provenance — tests/e2e_ingest_search.rs
ingest_url_then_search_finds_it_with_webpage_provenance — tests/e2e_ingest_search.rs
credential_never_reaches_the_stored_corpus — tests/e2e_ingest_transcripts.rs
emptied_session_is_tombstoned_not_unchanged — tests/e2e_ingest_transcripts.rs
excluding_a_project_purges_its_already_indexed_sessions — tests/e2e_ingest_transcripts.rs
full_reembeds_under_a_new_model — tests/e2e_ingest_transcripts.rs
ingest_twice_then_grow_embeds_exactly_the_new_turns — tests/e2e_ingest_transcripts.rs
research_ingest_report_counts_are_exact — tests/e2e_ingest_transcripts.rs
survey_measures_without_embedding — tests/e2e_ingest_transcripts.rs
a_transcript_with_no_assistant_turn_stages_nothing — tests/hook_turn_live.rs
the_hook_stages_the_last_turn_reads_it_and_runs_twice_for_nothing — tests/hook_turn_live.rs
book_to_doc_maps_identity_and_provenance — src/book_ingest.rs
empty_fields_are_skipped_not_rendered_blank — src/book_ingest.rs
search_text_carries_subject_author_isbn_location — src/book_ingest.rs
alternatives_default_is_empty_array — src/cli.rs
alternatives_rejects_malformed_json_and_wrong_shape — src/cli.rs
alternatives_valid_shape_passes_through — src/cli.rs
artifacts_must_be_a_json_array — src/cli.rs
cli_definition_conforms — src/cli.rs
decision_add_requires_the_four_core_fields — src/cli.rs
decision_supersede_parses_uuids — src/cli.rs
envelope_carries_schema_version_1 — src/cli.rs
ingest_books_defaults_and_flags_parse — src/cli.rs
ingest_defaults_and_flags_parse — src/cli.rs
ingest_url_defaults_and_flags_parse — src/cli.rs
ingest_url_rejects_a_malformed_url — src/cli.rs
mv_parses_positionals — src/cli.rs
search_parses_repeatable_corpus_and_limit — src/cli.rs
decision_render_is_not_raw_json — src/federate.rs
_… 57 more_

Documentation coverage

Measure	Documented	Total
Public items with rustdoc	100	102
Public modules with a `//!` block	12	12

pie showData
    title Public items with rustdoc
    "Documented" : 100
    "No rustdoc detected" : 2

Metrics

Metric	Value
Rust source files	14
Source lines	6057
Code lines	4784
Public API items	102
Public modules	12
Tests	87
Examples	0
Cargo features	1
Direct runtime dependencies	26
Workspace reverse dependencies	0

pie showData
    title Public API by kind
    "constant" : 14
    "enum" : 13
    "function" : 51
    "method" : 3
    "struct" : 21

pie showData
    title Rust source composition
    "Code" : 4784
    "Blank or comment" : 1273

Generation

Rendered by tools-corpus corpus readme from repository evidence alone, renderer schema 2, lexicon current. No model, network service or database was consulted. Regenerate with tools-corpus corpus readme --write; verify with --check.

All application · Manual