infrastructure-web-ingest

Composed website-content ingester: fetch (infrastructure-fetcher -- SSRF-hardened, rate-limited, sha256-deduped, preserve-before-parse) -> decode/extract/chunk (infrastructure-html-text) -> transactional index (infrastructure-knowledge-index), with stored-artifact identity in the report for replay/re-extract without refetch. Non-goals (v1): crawling/link-following, JS rendering, robots.txt policy, PDF/media ingestion -- HTML only.

What it is for

infrastructure-web-ingest -- composed website-content ingester.

Thin orchestration over five existing layer crates: infrastructure-fetcher (fetch, SSRF-hardened, rate-limited, sha256-deduped, content-addressed storage), infrastructure-html-text (decode/extract/chunk), and infrastructure-knowledge-index (transactional index write). See pipeline::ingest_url for the composed entry point and docs/userstories/sprint-3.24-web-ingest.md for the full design + Gate 1.5 consensus log.

Non-goals (v1)

is a different mechanism (frontier, politeness, dedup-at-scale) layered ON TOP of this, not inside it.

• JS rendering. infrastructure-fetcher's BridgeExecutor exists for

sites that need a real browser, but this crate does not select it -- FetchHints::html_page()'s ExecutorPreference::Auto may still promote to it on a bot-challenge retry, but no code here targets rendering.

• `robots.txt` policy. This is an operator-directed single fetch
• Classification / keyword extraction. Extension seams for later, not

v1 scope.

• Scheduled refresh. Every call is a fresh, explicit operator action.
• PDF / non-HTML media. text/html and application/xhtml+xml only

-- anything else is WebIngestError::NotHtml.

Concurrency contract

Two concurrent ingest_url calls for the SAME (tenant, corpus, source_ref) are last-write-wins: IndexService::index_doc's per-doc advisory lock (keyed by (tenant, corpus, source_ref)) plus its single transaction guarantee no torn write -- one call's full doc+chunks commit atomically before or after the other's, never interleaved. Neither call is told it lost the race; whichever committed last is what search sees until the next ingest. This is a deliberate v1 simplification (Gate 1.5 round 2, unanimous): a fetch-generation recheck is DEFERRED until a scheduled/concurrent ingester actually exists to need it. Self-healing: the next ingest_url call re-establishes the current state regardless of what a prior race left behind.

Capabilities

pipeline (other)

The composed pipeline: fetch -> decode/extract/chunk -> index.

WebIngestConfig

The composed pipeline: fetch -> decode/extract/chunk -> index.

WebIngestError

The composed pipeline: fetch -> decode/extract/chunk -> index.

WebIngestReport

The composed pipeline: fetch -> decode/extract/chunk -> index.

TitleSource

Naming a fetched document (B-079).

How to use it

No examples/ target and no doctest in this crate's rustdoc. The tests listed under Verification are the closest executable usage.

Module structure

infrastructure_web_ingest

• pipeline
• title

flowchart TD
  n_infrastructure_web_ingest["infrastructure_web_ingest"]
  n_infrastructure_web_ingest --> n_pipeline["pipeline"]
  n_infrastructure_web_ingest --> n_title["title"]

Public surface

`pipeline`

`title`

Re-exports. Exported here, defined elsewhere.

Boundary

Depends on no other workspace tier.

Shares tier infrastructure with 82 other crates: infrastructure-acquire, infrastructure-adapters-google-calendar, infrastructure-adapters-google-gmail, infrastructure-adapters-google-places, infrastructure-adapters-google-trends, infrastructure-adapters-shodan, infrastructure-adapters-yelp, infrastructure-agent, … (82 total).

_What this crate deliberately does NOT own is a judgment. No committed registry records one for it, so none is stated here._

Where it sits

Dependencies

Runtime, in this workspace.

Runtime, from outside the workspace.

Development, in this workspace.

Development, from outside the workspace.

Build. None.

Depended on by. 1 workspace crate.

Signal flow — what reaches this crate, and what it reaches.

flowchart LR
  n_application_wiki["application-wiki"] -->|uses| SELF
  SELF["infrastructure-web-ingest"]
  SELF -->|development| n_infrastructure_fetcher["infrastructure-fetcher"]
  SELF -->|development| n_operations_rate_limit["operations-rate-limit"]
  SELF -->|runtime| n_infrastructure_embeddings["infrastructure-embeddings"]
  SELF -->|runtime| n_infrastructure_fetcher["infrastructure-fetcher"]
  SELF -->|runtime| n_infrastructure_html_text["infrastructure-html-text"]
  SELF -->|runtime| n_infrastructure_knowledge_index["infrastructure-knowledge-index"]
  SELF -->|runtime| n_infrastructure_storage["infrastructure-storage"]
  classDef self fill:#1f883d,stroke:#1f883d,color:#fff;
  class SELF self;

Feature flags

No Cargo features are defined: every capability is unconditional, so no consumer can receive a half-wired crate.

Targets

Error model

Operational characteristics

No unsafe block, unsafe fn, unsafe impl or unsafe trait was found by the parser anywhere in this crate's source.

Configuration

No environment variable is read with a literal name anywhere in this crate. A variable whose key is computed at run time cannot be listed here, and is not claimed to be absent.

Related capabilities

1 workspace crate depends on this one: application-wiki.

Verification

Evidence by module. How often each public module is named by something executable.

What the tests establish, by name:

• happy_path_then_dedup_writes_nothing_new — tests/e2e_db.rs
• source_ref_and_source_kind_match_the_federate_ledger_precedent — tests/e2e_db.rs
• byte_cap_applies_to_wire_bytes_not_decompressed_bytes — tests/pipeline_offline.rs
• cap_exceeded_surfaces_as_a_fetch_error — tests/pipeline_offline.rs
• empty_extraction_when_the_page_is_all_chrome — tests/pipeline_offline.rs
• non_html_content_type_is_rejected_before_storage — tests/pipeline_offline.rs
• report_serde_round_trips_via_a_real_happy_path_shape — tests/pipeline_offline.rs
• ssrf_blocked_host_surfaces_the_underlying_error — tests/pipeline_offline.rs
• default_config_matches_documented_defaults — src/pipeline.rs
• is_acceptable_html_mime_accepts_html_and_xhtml_only — src/pipeline.rs
• report_round_trips_through_json — src/pipeline.rs
• strip_identity_is_a_no_op_for_a_clean_url — src/pipeline.rs
• strip_identity_removes_userinfo_and_fragment_keeps_query — src/pipeline.rs
• blank_inputs_are_treated_as_absent — src/title.rs
• first_h1_is_third — src/title.rs
• host_names_a_root_document — src/title.rs
• og_title_is_second — src/title.rs
• title_element_wins_when_present — src/title.rs
• title_source_serializes_snake_case_and_matches_as_str — src/title.rs
• trailing_slash_and_query_do_not_hide_the_segment — src/title.rs
• url_last_path_segment_is_the_final_fallback — src/title.rs

Documentation coverage

pie showData
    title Public items with rustdoc
    "Documented" : 9
    "No rustdoc detected" : 1

Metrics

pie showData
    title Public API by kind
    "constant" : 2
    "enum" : 2
    "function" : 2
    "method" : 2
    "struct" : 2

pie showData
    title Rust source composition
    "Code" : 390
    "Blank or comment" : 281

Generation

Rendered by tools-corpus corpus readme from repository evidence alone, renderer schema 2, lexicon current. No model, network service or database was consulted. Regenerate with tools-corpus corpus readme --write; verify with --check.