Local, deterministic structural decomposition of general markdown prose (memos, briefs, narrative documents -- anything with neither a table nor speaker-turn dialogue): heading-bounded sections and per-paragraph structural facts (word count, question-ness, repeated-citation-phrase references), emitting an opaque-ID shard. Section headings and paragraph body text never appear in the emitted shard -- only heading LEVEL (structure, not content) and per-paragraph statistics. Completes the document-structural-decomposition family alongside domain-table-shard and domain-transcript-shard (ADR 0035). Pure logic, zero I/O beyond reading the input file, zero AI calls.
| Tier | domain |
| Role | unclassified (baselined) |
| Path | crates/domain/prose-shard |
| Edition | 2021 |
| Targets | prose-shard, domain_prose_shard |
| Public items | 5 across 0 modules |
| Tests | 9 |
What it is for
Local, deterministic structural decomposition of general markdown prose: heading-bounded sections and per-paragraph structural facts (word count, question-ness, repeated-citation-phrase references), emitting an opaque-ID shard. Completes the document-structural- decomposition family alongside domain-table-shard (tabular data) and domain-transcript-shard (speaker-turn dialogue) -- see ADR 0035. Nothing here calls an AI or makes a network request.
Deliberately no per-paragraph type taxonomy (unlike transcript-shard's StatementType): that enum was grounded in a real transcript's actual content during design. Prose has no equivalent grounded fixture to derive categories from, and inventing one here would be exactly the kind of confident-but-ungrounded guess this whole family exists to avoid in the AI it hands the shard to -- so this crate reports only what it can compute without guessing: counts and a yes/no question-mark check, not an invented "this paragraph is an argument" label.
Section headings are not disclosed, unlike table column headers in table-shard. That crate's own doc comment flags header-as-metadata exposure as an open, unguarded risk; this crate closes it for its own case by keeping only the heading level (a structural fact -- how deep in the document) and dropping the heading text entirely.
Capabilities
DocumentShard
Local, deterministic structural decomposition of general markdown
| Item |
|---|
pub struct DocumentShard |
fn build_shard(text : & str, task : & str, allowed_outputs : Vec <String>) -> DocumentShard |
async fn build_shard_with_clusters(text : & str, task : & str, allowed_outputs : Vec <String>, embedder : & dyn infrastructure_embeddings::Embedder, similarity_threshold : f32,) -> Result <DocumentShard, infrastructure_embeddings::EmbeddingError> |
ParagraphShard
Local, deterministic structural decomposition of general markdown
| Item |
|---|
pub struct ParagraphShard |
SectionShard
Local, deterministic structural decomposition of general markdown
| Item |
|---|
pub struct SectionShard |
How to use it
No examples/ target and no doctest in this crate's rustdoc. The tests listed under Verification are the closest executable usage.
Module structure
No public modules: the crate root is its whole surface.
Public surface
`crate root`
| Item | What it is |
|---|---|
pub struct SectionShard | One heading-bounded section's structural facts |
pub struct ParagraphShard | One paragraph's structural facts |
pub struct DocumentShard | The minimal-disclosure document shard: section shape, per-paragraph statistics, opaque repeated-phrase IDs, and the fixed question being asked |
fn build_shard(text : & str, task : & str, allowed_outputs : Vec <String>) -> DocumentShard | Build a document shard from text: detect repeated Title-Case phrases across every paragraph, assign each an opaque ID in first-appearance order, and reduce every paragraph to its structural facts |
async fn build_shard_with_clusters(text : & str, task : & str, allowed_outputs : Vec <String>, embedder : & dyn infrastructure_embeddings::Embedder, similarity_threshold : f32,) -> Result <DocumentShard, infrastructure_embeddings::EmbeddingError> | Same as build_shard, plus embedding-based paragraph clustering (foundation-text-clustering): each paragraph's cluster field is set to an opaque, zero-based label grouping it with other paragraphs of similar embedding |
No pub use re-exports: every item above is declared in this crate.
Boundary
Reaches into foundation, infrastructure.
Shares tier domain with 41 other crates: domain-agreements, domain-ai-report, domain-billing, domain-catalog, domain-classify, domain-comments, domain-competitive-intel, domain-contact, … (41 total).
_What this crate deliberately does NOT own is a judgment. No committed registry records one for it, so none is stated here._
Where it sits
| Tier (ontology) | domain |
| Architectural role (taxonomy) | unclassified (baselined) |
| Location | crates/domain/prose-shard |
| Vocabulary in force (lexicon) | current |
Tier flow. Which tiers this crate's own edges cross.
flowchart LR n_domain["domain"] --> n_foundation["foundation"] n_domain["domain"] --> n_infrastructure["infrastructure"]
Dependencies
Runtime, in this workspace.
| Crate | Tier | Optional | Only on |
|---|---|---|---|
| `domain-transcript-shard` | domain | no | always |
| `foundation-text-clustering` | foundation | no | always |
| `foundation-text-signals` | foundation | no | always |
| `infrastructure-embeddings` | infrastructure | no | always |
Runtime, from outside the workspace.
| Crate | Requirement | Features | Optional | Only on |
|---|---|---|---|---|
clap | ^4 | derive | no | always |
pulldown-cmark | ^0.10 | — | no | always |
serde | ^1 | derive, derive | no | always |
serde_json | ^1 | — | no | always |
tokio | ^1 | full, macros, rt-multi-thread | no | always |
Development, in this workspace.
| Crate | Tier | Optional | Only on |
|---|---|---|---|
| `tools-cli-conformance` | tools | no | always |
Build. None.
Depended on by. 2 workspace crates.
Signal flow — what reaches this crate, and what it reaches.
flowchart LR n_tools_doc_ask["tools-doc-ask"] -->|uses| SELF n_tools_knowitall["tools-knowitall"] -->|uses| SELF SELF["domain-prose-shard"] SELF -->|development| n_tools_cli_conformance["tools-cli-conformance"] SELF -->|runtime| n_domain_transcript_shard["domain-transcript-shard"] SELF -->|runtime| n_foundation_text_clustering["foundation-text-clustering"] SELF -->|runtime| n_foundation_text_signals["foundation-text-signals"] SELF -->|runtime| n_infrastructure_embeddings["infrastructure-embeddings"] classDef self fill:#1f883d,stroke:#1f883d,color:#fff; class SELF self;
Feature flags
No Cargo features are defined: every capability is unconditional, so no consumer can receive a half-wired crate.
Targets
| Kind | Name | Source |
|---|---|---|
| bin | prose-shard | `src/main.rs` |
| lib | domain_prose_shard | `src/lib.rs` |
Error model
No public error type was detected: no public item declares a type named *Error, and no public signature returns one.
Operational characteristics
| Property | Evidence |
|---|---|
| async public surface | yes |
| async runtime | yes |
| database access | none detected |
| network I/O | none detected |
| unsafe code | none detected |
| environment variables | none detected |
No unsafe block, unsafe fn, unsafe impl or unsafe trait was found by the parser anywhere in this crate's source.
Configuration
No environment variable is read with a literal name anywhere in this crate. A variable whose key is computed at run time cannot be listed here, and is not claimed to be absent.
Related capabilities
2 workspace crates depend on this one: tools-doc-ask, tools-knowitall.
Verification
| Kind | Count |
|---|---|
| Unit tests | 9 |
| Integration tests | 0 |
| Examples | 0 |
| Doctests | 0 |
Evidence by module. How often each public module is named by something executable.
| Module | Tests | Examples | Consumers |
|---|---|---|---|
crate root | 5 | 0 | 3 |
What the tests establish, by name:
a_code_fence_is_not_misread_as_a_paragraph—src/lib.rsbuild_shard_detects_the_question_and_the_repeated_citation—src/lib.rsbuild_shard_never_leaks_real_heading_or_paragraph_text—src/lib.rsbuild_shard_populates_real_sentiment_and_pos_signals—src/lib.rsbuild_shard_with_clusters_populates_every_paragraphs_cluster_field—src/lib.rsbuild_shard_without_clustering_leaves_cluster_field_none—src/lib.rsempty_document_yields_one_empty_root_section_and_no_paragraphs—src/lib.rsparse_structure_finds_the_right_sections_and_paragraph_shape—src/lib.rscli_definition_conforms—src/main.rs
Documentation coverage
| Measure | Documented | Total |
|---|---|---|
| Public items with rustdoc | 5 | 5 |
Public modules with a //! block | 0 | 0 |
pie showData
title Public items with rustdoc
"Documented" : 5
"No rustdoc detected" : 0
Metrics
| Metric | Value |
|---|---|
| Rust source files | 2 |
| Source lines | 480 |
| Code lines | 327 |
| Public API items | 5 |
| Public modules | 0 |
| Tests | 9 |
| Examples | 0 |
| Cargo features | 0 |
| Direct runtime dependencies | 9 |
| Workspace reverse dependencies | 2 |
pie showData
title Public API by kind
"function" : 2
"struct" : 3
pie showData
title Rust source composition
"Code" : 327
"Blank or comment" : 153
Generation
Rendered by tools-corpus corpus readme from repository evidence alone, renderer schema 2, lexicon current. No model, network service or database was consulted. Regenerate with tools-corpus corpus readme --write; verify with --check.