infrastructure-html-text

Pure, sync HTML->text extraction mechanism: charset-aware decode (encoding_rs; declared charset -> <meta charset> sniff -> windows-1252 fallback, flags recorded), bounded-output structural extraction via lol_html (drops script/style/nav/svg/noscript/template/iframe + comments, outline-stack heading_path with skipped-level retention, output byte cap + per-heading char cap with per-class truncation flags), and section-aware chunking over infrastructure-embeddings::chunk_text with document-absolute char offsets taken from the chunker's returned spans. No I/O, no async, no lossy silent fallback -- the reusable extraction mechanism composed by infrastructure-web-ingest.

What it is for

infrastructure-html-text -- pure, synchronous HTML -> text extraction.

No I/O, no async, no network: bytes and config in, structured text out. Three stages, each independently testable and each mirroring an existing in-repo convention rather than inventing a new one:

1. decode -- bytes -> String via encoding_rs: declared charset -> <meta charset> sniff -> windows-1252 fallback, with a decode::DecodeReport so a lossy decode is never silent. 2. extract -- bounded-output HTML -> text via lol_html (the same streaming rewriter infrastructure-dom-reduce uses): drops script/style/nav/svg/noscript/template/iframe subtrees and comments, builds an outline-stack heading_path from h1-h6, and caps both total output and any single heading's length. 3. chunker -- section-aware chunking over infrastructure_embeddings::chunk_text, the same chunker tools-org-knowledge's markdown pipeline uses, with document-absolute char offsets read directly from the chunker's returned spans.

The reusable mechanism consumed by infrastructure-web-ingest, which adds fetch, storage, and the knowledge-index write.

Capabilities

ChunkInput

Section-aware chunking over infrastructure_embeddings::chunk_text.

DecodeReport

bytes -> String decode: declared charset -> <meta charset> sniff ->

extract (other)

Bounded-output HTML -> text extraction via lol_html.

ExtractConfig

Bounded-output HTML -> text extraction via lol_html.

Extraction

Bounded-output HTML -> text extraction via lol_html.

Section

Bounded-output HTML -> text extraction via lol_html.

TruncationFlags

Bounded-output HTML -> text extraction via lol_html.

How to use it

No examples/ target and no doctest in this crate's rustdoc. The tests listed under Verification are the closest executable usage.

Module structure

infrastructure_html_text

• chunker
• decode
• extract

flowchart TD
  n_infrastructure_html_text["infrastructure_html_text"]
  n_infrastructure_html_text --> n_chunker["chunker"]
  n_infrastructure_html_text --> n_decode["decode"]
  n_infrastructure_html_text --> n_extract["extract"]

Public surface

`chunker`

`decode`

`extract`

Re-exports. Exported here, defined elsewhere.

Boundary

Depends on no other workspace tier.

Shares tier infrastructure with 82 other crates: infrastructure-acquire, infrastructure-adapters-google-calendar, infrastructure-adapters-google-gmail, infrastructure-adapters-google-places, infrastructure-adapters-google-trends, infrastructure-adapters-shodan, infrastructure-adapters-yelp, infrastructure-agent, … (82 total).

_What this crate deliberately does NOT own is a judgment. No committed registry records one for it, so none is stated here._

Where it sits

Dependencies

Runtime, in this workspace.

Runtime, from outside the workspace.

Development. None.

Build. None.

Depended on by. 4 workspace crates.

Signal flow — what reaches this crate, and what it reaches.

flowchart LR
  n_application_party_discovery["application-party-discovery"] -->|uses| SELF
  n_infrastructure_acquire["infrastructure-acquire"] -->|uses| SELF
  n_infrastructure_web_contact_extractor["infrastructure-web-contact-extractor"] -->|uses| SELF
  n_infrastructure_web_ingest["infrastructure-web-ingest"] -->|uses| SELF
  SELF["infrastructure-html-text"]
  SELF -->|runtime| n_infrastructure_embeddings["infrastructure-embeddings"]
  classDef self fill:#1f883d,stroke:#1f883d,color:#fff;
  class SELF self;

Feature flags

No Cargo features are defined: every capability is unconditional, so no consumer can receive a half-wired crate.

Targets

Error model

No public error type was detected: no public item declares a type named *Error, and no public signature returns one.

Operational characteristics

No unsafe block, unsafe fn, unsafe impl or unsafe trait was found by the parser anywhere in this crate's source.

Configuration

No environment variable is read with a literal name anywhere in this crate. A variable whose key is computed at run time cannot be listed here, and is not claimed to be absent.

Related capabilities

4 workspace crates depend on this one: application-party-discovery, infrastructure-acquire, infrastructure-web-contact-extractor, infrastructure-web-ingest.

Verification

Evidence by module. How often each public module is named by something executable.

What the tests establish, by name:

• a_section_of_only_whitespace_produces_no_chunks — src/chunker.rs
• adr_0016_defaults_are_2000_320 — src/chunker.rs
• empty_sections_produce_no_chunks — src/chunker.rs
• heading_path_propagates_from_the_owning_section — src/chunker.rs
• offsets_are_document_absolute_across_section_boundaries — src/chunker.rs
• spans_come_from_the_chunker_never_from_searching_the_document — src/chunker.rs
• unicode_content_offsets_are_char_based_not_byte_based — src/chunker.rs
• contradictory_meta_loses_to_declared_charset — src/decode.rs
• declared_windows_1252_decodes_high_bytes_correctly — src/decode.rs
• empty_input_decodes_to_empty_string_without_errors — src/decode.rs
• invalid_utf8_bytes_with_declared_utf8_sets_had_errors — src/decode.rs
• meta_charset_sniff_used_when_no_declared_charset — src/decode.rs
• meta_http_equiv_content_type_charset_is_sniffed — src/decode.rs
• no_charset_information_falls_back_to_windows_1252 — src/decode.rs
• unrecognized_declared_charset_falls_through_to_meta_sniff — src/decode.rs
• utf8_bytes_decode_as_utf8_with_no_errors — src/decode.rs
• blank_title_is_none — src/extract.rs
• comments_are_dropped — src/extract.rs
• empty_input_yields_empty_extraction — src/extract.rs
• first_h1_is_captured_normalized_and_skips_dropped_subtrees — src/extract.rs
• first_h1_is_none_when_only_lower_headings_exist — src/extract.rs
• head_without_title_element_yields_none — src/extract.rs
• heading_inside_dropped_subtree_does_not_pollute_the_outline — src/extract.rs
• heading_path_nests_and_pops_on_sibling — src/extract.rs
• heading_text_is_included_in_its_own_section — src/extract.rs
• max_heading_chars_caps_a_single_heading_and_reports_it — src/extract.rs
• max_output_bytes_caps_extraction_and_reports_it — src/extract.rs
• mixed_self_closed_and_properly_closed_drop_elements — src/extract.rs
• nested_inline_markup_inside_a_heading_concatenates — src/extract.rs
• no_headings_at_all_yields_one_preamble_section — src/extract.rs
• _… 13 more_

Documentation coverage

pie showData
    title Public items with rustdoc
    "Documented" : 9
    "No rustdoc detected" : 1

Metrics

pie showData
    title Public API by kind
    "function" : 3
    "method" : 1
    "struct" : 6

pie showData
    title Rust source composition
    "Code" : 779
    "Blank or comment" : 357

Generation

Rendered by tools-corpus corpus readme from repository evidence alone, renderer schema 2, lexicon current. No model, network service or database was consulted. Regenerate with tools-corpus corpus readme --write; verify with --check.