infrastructure tier

`infrastructure-doc-ingest`

Document ingest shared by the wiki index and the source-code corpus: markdown front-matter/heading parsing and chunking, rename planning from content hashes, source-kind classification, git provenance, and database row counts

Document ingest shared by the wiki index and the source-code corpus: markdown front-matter/heading parsing and chunking, rename planning from content hashes, source-kind classification, git provenance, and database row counts

Tier	infrastructure
Role	unclassified (baselined)
Path	`crates/infrastructure/doc-ingest`
Edition	2021
Targets	infrastructure_doc_ingest
Public items	33 across 5 modules
Tests	42

What it is for

Document-ingest library crates shared by the wiki index and the source-code corpus (sprint 4.21).

These lived in tools-org-knowledge because the wiki was written first. When the source-code corpus moved out to tools-corpus, both consumers needed them — and a shared library crate belongs below both, not inside one.

markdown: YAML front matter, ATX-heading pre-split with

file-absolute offsets, per-section chunking.

walk_plan: rename/removal planning from indexed-vs-on-disk hashes,

source-kind classification, title derivation, git provenance.

db_stats: row counts read from the database rather than counted by

hand (rule 14).

Capabilities

`db_stats (other)`

Generic per-table row counting, shared across whichever database a

Item
`async fn table_row_counts(pool : & PgPool, tables : & & str,) -> Result <Vec <(String, i64)>, sqlx::Error>`

`embedder (other)`

Adapter wiring infrastructure_embeddings::Embedder into

Item
`pub const MODEL_DIR_ENV: & str`

`AnyEmbedder`

Adapter wiring infrastructure_embeddings::Embedder into

Item
`pub enum AnyEmbedder`
`AnyEmbedder :: fn model_id(& self) -> & str`
`AnyEmbedder :: fn dimensions(& self) -> usize`
`AnyEmbedder :: async fn embed_batch(& self, texts : & String) -> Result <Vec <Vec <f32>>, EmbedError>`
`fn embedder_from_env() -> Result <AnyEmbedder, String>`

`KnowledgeEmbedderAdapter`

Adapter wiring infrastructure_embeddings::Embedder into

Item
`pub struct KnowledgeEmbedderAdapter<E : Embedder>`
`fn default_embedder() -> KnowledgeEmbedderAdapter <DeterministicEmbedder>`

`KnowledgeEmbedderAdapter<E>`

Adapter wiring infrastructure_embeddings::Embedder into

Item
`KnowledgeEmbedderAdapter<E> :: fn new(inner : E) -> Self`
`KnowledgeEmbedderAdapter<E> :: fn model_id(& self) -> & str`
`KnowledgeEmbedderAdapter<E> :: fn dimensions(& self) -> usize`
`KnowledgeEmbedderAdapter<E> :: async fn embed_batch(& self, texts : & String) -> Result <Vec <Vec <f32>>, EmbedError>`

`markdown (other)`

Markdown structure module (Sprint 3.4 T2).

Item
`fn parse_document(text : & str) -> Result <ParsedDoc, MarkdownError>`
`fn chunk_document(text : & str, doc : & ParsedDoc, opts : & ChunkOptions) -> Vec <DocChunk>`
`fn parse_and_chunk(text : & str, opts : & ChunkOptions,) -> Result <(ParsedDoc, Vec <DocChunk>), MarkdownError>`

`DocChunk`

Markdown structure module (Sprint 3.4 T2).

Item
`pub struct DocChunk`
`DocChunk :: fn to_chunk_input(& self) -> infrastructure_knowledge_index::ChunkInput`

`MarkdownError`

Markdown structure module (Sprint 3.4 T2).

Item
`pub enum MarkdownError`

`ParsedDoc`

Markdown structure module (Sprint 3.4 T2).

Item
`pub struct ParsedDoc`

`Section`

Markdown structure module (Sprint 3.4 T2).

Item
`pub struct Section`

`session_walk (other)`

Screened discovery of agent-session transcripts (sprint 4.21 stage 3).

Item
`fn project_is_excluded(project_dir : & str, excludes : & String) -> bool`
`fn walk_sessions(root : & Path, excludes : & String,) -> Result <Vec <SessionFile>, SessionWalkError>`

`SessionFile`

Screened discovery of agent-session transcripts (sprint 4.21 stage 3).

Item
`pub struct SessionFile`

`SessionWalkError`

Screened discovery of agent-session transcripts (sprint 4.21 stage 3).

Item
`pub enum SessionWalkError`

`walk_plan (other)`

Shared document-ingest library crates (sprint 4.21).

Item
`fn classify_source_kind(rel_ref : & str) -> & 'static str`
`fn derive_title(front_matter : & serde_json::Value, doc : & markdown::ParsedDoc) -> Option <String>`
`fn derive_title_or_file_name(front_matter : & serde_json::Value, doc : & markdown::ParsedDoc, rel_path : & str,) -> String`
`fn file_name_title(rel_path : & str) -> String`

`GitProvenance`

Shared document-ingest library crates (sprint 4.21).

Item
`pub struct GitProvenance`
`fn git_provenance(root : & Path) -> GitProvenance`

`WalkActions`

Shared document-ingest library crates (sprint 4.21).

Item
`pub struct WalkActions`
`fn plan_walk_actions(indexed : & BTreeMap <String, String>, on_disk : & BTreeMap <String, String>,) -> WalkActions`

How to use it

No examples/ target and no doctest in this crate's rustdoc. The tests listed under Verification are the closest executable usage.

Module structure

infrastructure_doc_ingest

db_stats
embedder
markdown
session_walk
walk_plan

flowchart TD
  n_infrastructure_doc_ingest["infrastructure_doc_ingest"]
  n_infrastructure_doc_ingest --> n_db_stats["db_stats"]
  n_infrastructure_doc_ingest --> n_embedder["embedder"]
  n_infrastructure_doc_ingest --> n_markdown["markdown"]
  n_infrastructure_doc_ingest --> n_session_walk["session_walk"]
  n_infrastructure_doc_ingest --> n_walk_plan["walk_plan"]

Public surface

`db_stats`

Item	What it is
`async fn table_row_counts(pool : & PgPool, tables : & & str,) -> Result <Vec <(String, i64)>, sqlx::Error>`	`(table, row_count)` for every name in `tables`, in the order given

`embedder`

Item	What it is
`pub struct KnowledgeEmbedderAdapter<E : Embedder>`	Wraps any `infrastructure_embeddings::Embedder` as a `KnowledgeEmbedder`.
`KnowledgeEmbedderAdapter<E> :: fn new(inner : E) -> Self`	Wrap an embedder implementation.
`fn default_embedder() -> KnowledgeEmbedderAdapter <DeterministicEmbedder>`	The dev/test default: deterministic hashed-feature vectors at the index's declared `VECTOR_DIM` (1024).
`pub const MODEL_DIR_ENV: & str`	Env var naming the ONNX BGE-m3 model directory
`pub enum AnyEmbedder`	Runtime embedder selection — one choice for ALL subcommands, so ingest and search always agree on the model (the index refuses mixed-model corpora).
`AnyEmbedder :: fn model_id(& self) -> & str`	—
`AnyEmbedder :: fn dimensions(& self) -> usize`	—
`AnyEmbedder :: async fn embed_batch(& self, texts : & String) -> Result <Vec <Vec <f32>>, EmbedError>`	—
`fn embedder_from_env() -> Result <AnyEmbedder, String>`	Select the embedder from the environment (Gate 1.5 decision 5, sprint 3.15): `MODEL_DIR_ENV` set and the `ort` feature built → local ONNX BGE-m3; otherwise deterministic dev vectors with a stderr notice
`KnowledgeEmbedderAdapter<E> :: fn model_id(& self) -> & str`	—
`KnowledgeEmbedderAdapter<E> :: fn dimensions(& self) -> usize`	—
`KnowledgeEmbedderAdapter<E> :: async fn embed_batch(& self, texts : & String) -> Result <Vec <Vec <f32>>, EmbedError>`	—

`markdown`

Item	What it is
`pub enum MarkdownError`	Parse failures worth failing a file over
`pub struct Section`	A contiguous run of the document under one heading trail
`pub struct DocChunk`	One embeddable chunk with file-absolute provenance.
`DocChunk :: fn to_chunk_input(& self) -> infrastructure_knowledge_index::ChunkInput`	Convert to the knowledge-index boundary shape (T3 ingest uses this).
`pub struct ParsedDoc`	Structural parse result: front matter + heading-split sections.
`fn parse_document(text : & str) -> Result <ParsedDoc, MarkdownError>`	Extract front matter and pre-split the body on ATX headings
`fn chunk_document(text : & str, doc : & ParsedDoc, opts : & ChunkOptions) -> Vec <DocChunk>`	Chunk every section of an already-parsed doc; offsets are file-absolute.
`fn parse_and_chunk(text : & str, opts : & ChunkOptions,) -> Result <(ParsedDoc, Vec <DocChunk>), MarkdownError>`	Convenience: parse + chunk in one call

`session_walk`

Item	What it is
`pub struct SessionFile`	One discovered session file.
`pub enum SessionWalkError`	Walk failures
`fn project_is_excluded(project_dir : & str, excludes : & String) -> bool`	True when a project directory name matches any exclude pattern (case-insensitive substring — operator decision 2026-08-08: case-file projects stay out of the corpus, including future ones).
`fn walk_sessions(root : & Path, excludes : & String,) -> Result <Vec <SessionFile>, SessionWalkError>`	Discover session files under `root`, recursively (subagent/workflow transcripts live at `<project>/<session-uuid>/subagents/*/.jsonl`)

`walk_plan`

Item	What it is
`pub struct WalkActions`	What the walk diff decided (pure; applied by the caller's ingest run).
`fn plan_walk_actions(indexed : & BTreeMap <String, String>, on_disk : & BTreeMap <String, String>,) -> WalkActions`	Plan rename/removal actions from the indexed state vs the on-disk state
`fn classify_source_kind(rel_ref : & str) -> & 'static str`	Classify a repo-relative markdown path into a `source_kind` ('adr' \
`fn derive_title(front_matter : & serde_json::Value, doc : & markdown::ParsedDoc) -> Option <String>`	Derive a doc title: front matter `title` first, else the first ATX heading.
`fn derive_title_or_file_name(front_matter : & serde_json::Value, doc : & markdown::ParsedDoc, rel_path : & str,) -> String`	`derive_title`, then the file name when the document names itself nowhere (B-079: `knowledge_doc.title` is NOT NULL and non-blank, so a file-backed doc with no front-matter title and no heading is named by its path's last segment -- the same rule as knowledge-index migration 0003's backfill and `infrastructure-web-ingest`'s URL-segment arm).
`fn file_name_title(rel_path : & str) -> String`	The last non-empty `/`-separated segment of `rel_path`; the whole string when there is none (blank or slash-only)
`pub struct GitProvenance`	Git provenance stamped on every doc at ingest (memo step 8).
`fn git_provenance(root : & Path) -> GitProvenance`	Read git provenance for `root` via strict argument-vector `git` calls (never a shell)

No pub use re-exports: every item above is declared in this crate.

Boundary

Reaches into foundation.

Shares tier infrastructure with 82 other crates: infrastructure-acquire, infrastructure-adapters-google-calendar, infrastructure-adapters-google-gmail, infrastructure-adapters-google-places, infrastructure-adapters-google-trends, infrastructure-adapters-shodan, infrastructure-adapters-yelp, infrastructure-agent, … (82 total).

_What this crate deliberately does NOT own is a judgment. No committed registry records one for it, so none is stated here._

Where it sits

Tier (ontology)	`infrastructure`
Architectural role (taxonomy)	`unclassified (baselined)`
Location	`crates/infrastructure/doc-ingest`
Vocabulary in force (lexicon)	`current`

Tier flow. Which tiers this crate's own edges cross.

flowchart LR
  n_infrastructure["infrastructure"] --> n_foundation["foundation"]

Dependencies

Runtime, in this workspace.

Crate	Tier	Optional	Only on
`foundation-bounded-io`	foundation	no	always
`infrastructure-embeddings`	infrastructure	no	always
`infrastructure-knowledge-index`	infrastructure	no	always

Runtime, from outside the workspace.

Crate	Requirement	Features	Optional	Only on
`async-trait`	`^0.1`	—	no	always
`serde`	`^1`	`derive`, `derive`	no	always
`serde_json`	`^1`	—	no	always
`serde_yaml`	`^0.9`	—	no	always
`sqlx`	`^0.8`	`runtime-tokio`, `postgres`, `chrono`, `uuid`, `json`	no	always
`thiserror`	`^2`	—	no	always

Development, from outside the workspace.

Crate	Requirement	Features	Optional	Only on
`tokio`	`^1`	`full`, `macros`, `rt-multi-thread`	no	always
`url`	`^2`	—	no	always

Build. None.

Depended on by. 3 workspace crates.

Signal flow — what reaches this crate, and what it reaches.

flowchart LR
  n_application_wiki["application-wiki"] -->|uses| SELF
  n_tools_corpus["tools-corpus"] -->|uses| SELF
  n_tools_retrieval_eval["tools-retrieval-eval"] -->|uses| SELF
  SELF["infrastructure-doc-ingest"]
  SELF -->|runtime| n_foundation_bounded_io["foundation-bounded-io"]
  SELF -->|runtime| n_infrastructure_embeddings["infrastructure-embeddings"]
  SELF -->|runtime| n_infrastructure_knowledge_index["infrastructure-knowledge-index"]
  classDef self fill:#1f883d,stroke:#1f883d,color:#fff;
  class SELF self;

Feature flags

Feature	Enables	On by default
`ort`	`infrastructure-embeddings/ort`	no

flowchart LR
  n_ort["ort"] --> n_infrastructure_embeddings_ort["infrastructure-embeddings/ort"]

Targets

Kind	Name	Source
lib	`infrastructure_doc_ingest`	`src/lib.rs`

Error model

Error type	Named by
`MarkdownError`	`parse_and_chunk`, `parse_document`
`SessionWalkError`	`walk_sessions`

Operational characteristics

Property	Evidence
async public surface	yes
async runtime	none detected
database access	yes
network I/O	none detected
unsafe code	none detected
environment variables	yes

No unsafe block, unsafe fn, unsafe impl or unsafe trait was found by the parser anywhere in this crate's source.

Configuration

Variable	Read in
`DATABASE_URL`	`src/db_stats.rs`
`ORG_KNOWLEDGE_STRICT_CUDA`	`src/embedder.rs`
`TEST_DATABASE_URL`	`src/db_stats.rs`

3 workspace crates depend on this one: application-wiki, tools-corpus, tools-retrieval-eval.

Verification

Kind	Count
Unit tests	42
Integration tests	0
Examples	0
Doctests	0

Evidence by module. How often each public module is named by something executable.

Module	Tests	Examples	Consumers
`db_stats`	1	0	2
`embedder`	5	0	5
`markdown`	7	0	11
`session_walk`	4	0	4
`walk_plan`	8	0	14

What the tests establish, by name:

table_row_counts_reports_every_named_table — src/db_stats.rs
any_embedder_enum_passes_through_exactly — src/embedder.rs
embed_batch_preserves_order_and_matches_query_path — src/embedder.rs
empty_batch_is_empty — src/embedder.rs
empty_model_dir_means_unset_never_a_load_attempt — src/embedder.rs
identity_and_dimensions_pass_through — src/embedder.rs
atx_edge_cases — src/markdown.rs
chunk_offsets_are_file_absolute_not_section_relative — src/markdown.rs
chunks_carry_their_section_heading_path — src/markdown.rs
empty_front_matter_block_is_empty_object — src/markdown.rs
fence_close_requires_same_char_and_at_least_open_length — src/markdown.rs
front_matter_extracted_as_json_object — src/markdown.rs
front_matter_offsets_are_file_absolute — src/markdown.rs
heading_path_nests_and_pops_on_sibling — src/markdown.rs
headings_inside_fenced_code_blocks_are_ignored — src/markdown.rs
invalid_yaml_front_matter_errors — src/markdown.rs
no_front_matter_yields_empty_object_and_zero_body_start — src/markdown.rs
non_mapping_front_matter_errors — src/markdown.rs
reslice_property_holds_for_every_chunk — src/markdown.rs
skipped_heading_levels_keep_trail — src/markdown.rs
to_chunk_input_maps_fields — src/markdown.rs
unclosed_fence_swallows_rest_of_doc — src/markdown.rs
unterminated_fence_is_thematic_break_not_front_matter — src/markdown.rs
whitespace_only_sections_produce_no_chunks — src/markdown.rs
a_missing_root_is_refused_not_silently_empty — src/session_walk.rs
excludes_match_case_insensitively_as_substrings — src/session_walk.rs
only_jsonl_below_the_root_is_discovered — src/session_walk.rs
ambiguous_many_to_many_is_delete_plus_create_with_warning — src/walk_plan.rs
derive_title_or_file_name_prefers_front_matter_then_heading_then_file_name — src/walk_plan.rs
diff_keys_partitions_by_presence — src/walk_plan.rs
_… 12 more_

Documentation coverage

Measure	Documented	Total
Public items with rustdoc	27	33
Public modules with a `//!` block	5	5

pie showData
    title Public items with rustdoc
    "Documented" : 27
    "No rustdoc detected" : 6

Metrics

Metric	Value
Rust source files	6
Source lines	1487
Code lines	1111
Public API items	33
Public modules	5
Tests	42
Examples	0
Cargo features	1
Direct runtime dependencies	9
Workspace reverse dependencies	3

pie showData
    title Public API by kind
    "constant" : 1
    "enum" : 3
    "function" : 14
    "method" : 8
    "struct" : 7

pie showData
    title Rust source composition
    "Code" : 1111
    "Blank or comment" : 376

Generation

Rendered by tools-corpus corpus readme from repository evidence alone, renderer schema 2, lexicon current. No model, network service or database was consulted. Regenerate with tools-corpus corpus readme --write; verify with --check.

All infrastructure · Manual