infrastructure-har — IMP FORGE

Bounded, streaming reader for HTTP Archive (.har) files. Yields entries one at a time via a staged visitor so a consumer decides from the REQUEST whether a response body is ever decoded — the entries array is never materialised. Explicit caps on entry size, body size, nesting depth, string length, entry count and wall-clock; secret redaction applied inside the reader. Knows nothing about any specific website (PROCESS §17).

Tier	infrastructure
Role	unclassified (baselined)
Path	`crates/infrastructure/har`
Edition	2021
Targets	push_sniffies, scan_hars, scan_sniffies, survey, infrastructure_har
Public items	52 across 1 module
Tests	15

What it is for

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

HAR is a public format: { "log": { "entries": { "request": …, "response": … } } }. This crate knows nothing about any particular website. Site-specific extraction is a consumer concern (see deep_scan for the generic walker consumers build on).

# Why a staged visitor rather than an iterator

The semantically important grouping in a captured session frequently lives in the request (which query was issued, which collection token), while the members live in the paired response. An extractor that sees only responses loses the grouping.

But a response body may be enormous — a measured real sample carries 115 MiB of base64 image bodies inside 1,423 string tokens. So the consumer must be able to decide from the request whether the response body is ever decoded.

EntryVisitor::on_request therefore runs first and returns an EntryAction. Only on EntryAction::Continue is a body produced, and ResponseSink::on_head gets a second refusal point once the MIME type is known (a SERP extractor wants text/html; a GraphQL extractor wants text/javascript — neither is knowable from the request alone).

An Iterator<Item = (Request, Response)> cannot express this: it must construct the whole item before yielding, so the body is decoded before the consumer can decline it.

# Bounds (PROCESS §17)

Every limit in Limits is explicit and caller-supplied. Exceeding one is an error, never a silent truncation — a partial import that looks complete is the failure mode this crate exists to prevent.

# Redaction

Limits::redact is applied inside the reader, so credential-bearing header values and body fragments never reach a consumer at all. Cheaper and safer than scrubbing downstream.

Capabilities

`crate root`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub const DEFAULT_MAX_ENTRY_BYTES: usize`
`pub const DEFAULT_MAX_BODY_BYTES: usize`
`pub const DEFAULT_MAX_REQUEST_BODY_BYTES: usize`
`pub const DEFAULT_MAX_ENTRIES: usize`
`pub const DEFAULT_DEADLINE: Duration`

`EntryAction`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub enum EntryAction<S>`

`EntryMeta`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub struct EntryMeta<'a>`

`EntryVisitor`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub trait EntryVisitor`

`HarError`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub enum HarError`

`HarReader`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub struct HarReader`
`HarReader :: fn new() -> Self`
`HarReader :: fn with_limits(limits : Limits) -> Self`
`HarReader :: fn read <R : BufRead, V : EntryVisitor>(& self, reader : R, visitor : & mut V,) -> Result <Result <ReadStats, V::Error>, HarError>`
`HarReader :: fn default() -> Self`

`HeadDecision`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub enum HeadDecision`

`HeaderView`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub struct HeaderView<'a>`

`Limits`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub struct Limits`
`Limits :: fn default() -> Self`
`Limits :: fn fmt(& self, f : & mut std::fmt::Formatter <'_>) -> std::fmt::Result`

`ReadStats`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub struct ReadStats`
`ReadStats :: fn body_ratio(& self) -> f64`
`ReadStats :: fn looks_incomplete(& self) -> bool`

`Redactor`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub type Redactor: Box <dyn Fn(& str) -> String + Send + Sync>`

`RequestView`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub struct RequestView<'a>`

`RequestView<'a>`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`RequestView<'a> :: fn parse_json <T : serde::de::DeserializeOwned>(& self) -> Result <T, serde_json::Error>`
`RequestView<'a> :: fn form_pairs(& self) -> impl Iterator <Item =(String, String)> + '_`
`RequestView<'a> :: fn form_value(& self, key : & str) -> Option <String>`

`ResponseBodyMode`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub enum ResponseBodyMode`

`ResponseHead`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub struct ResponseHead<'a>`

`ResponseHead<'_>`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`ResponseHead<'_> :: fn is_base64(& self) -> bool`
`ResponseHead<'_> :: fn mime_contains(& self, needle : & str) -> bool`

`ResponseOutcome`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub enum ResponseOutcome`

`ResponseSink`

infrastructure-har — a bounded, streaming reader for HTTP Archive (.har) files.

Item
`pub trait ResponseSink`

`scan (other)`

Generic recursive JSON descent, plus the two things that make captured social data linkable:

Item
`fn surname_index(names : & String) -> BTreeMap <String, Vec <String>>`

`HispanicName`

Generic recursive JSON descent, plus the two things that make captured social data linkable:

Item
`pub struct HispanicName`
`HispanicName :: fn parse(raw : & str) -> Self`
`HispanicName :: fn shared_line(& self, other : & Self) -> Option <Lineage>`

`Kin`

Generic recursive JSON descent, plus the two things that make captured social data linkable:

Item
`pub enum Kin`
`fn kin_lexicon() -> & 'static (& 'static str, Kin)`
`fn kin_in(text : & str) -> Vec <Kin>`

`Lineage`

Generic recursive JSON descent, plus the two things that make captured social data linkable:

Item
`pub enum Lineage`

`LinkBasis`

Generic recursive JSON descent, plus the two things that make captured social data linkable:

Item
`pub enum LinkBasis`

`ProposedLink`

Generic recursive JSON descent, plus the two things that make captured social data linkable:

Item
`pub struct ProposedLink`
`fn propose_surname_links(names : & String) -> Vec <ProposedLink>`

`ScanHit`

Generic recursive JSON descent, plus the two things that make captured social data linkable:

Item
`pub struct ScanHit`

`Scanner`

Generic recursive JSON descent, plus the two things that make captured social data linkable:

Item
`pub trait Scanner`
`fn deep_scan <S : Scanner>(value : & Value, scanner : & mut S, max_depth : usize)`

`Seen`

Generic recursive JSON descent, plus the two things that make captured social data linkable:

Item
`pub struct Seen`
`Seen :: fn new() -> Self`
`Seen :: fn first_time <'a, I : IntoIterator <Item = & 'a str>>(& mut self, parts : I) -> bool`
`Seen :: fn len(& self) -> usize`
`Seen :: fn is_empty(& self) -> bool`

How to use it

From `examples/push_sniffies.rs`:


use std::cell::RefCell;
use std::collections::BTreeMap;
use std::fs::File;
use std::io::BufReader;
use std::rc::Rc;

use infrastructure_har::{
    EntryAction, EntryMeta, EntryVisitor, HarReader, HeadDecision, RequestView, ResponseBodyMode,
    ResponseHead, ResponseOutcome, ResponseSink,
};
use serde::Deserialize;
use serde_json::{json, Value};

const BATCH_ITEM_CAP: usize = 400;

#[derive(Deserialize)]
struct MessagesResponse {

From `examples/scan_hars.rs`:


use std::cell::RefCell;
use std::collections::BTreeMap;
use std::fs::File;
use std::io::BufReader;
use std::rc::Rc;

use infrastructure_har::scan::{kin_in, surname_index, HispanicName, Kin, Lineage, Scanner, Seen};
use infrastructure_har::{
    deep_scan, EntryAction, EntryMeta, EntryVisitor, HarReader, HeadDecision, Limits, ReadStats,
    RequestView, ResponseBodyMode, ResponseHead, ResponseOutcome, ResponseSink,
};
use serde_json::{Map, Value};

#[derive(Default)]
struct Findings {
    names: BTreeMap<String, usize>,
    comments: Vec<(Option<i64>, String, String)>,

Module structure

infrastructure_har

scan

Public surface

`crate root`

Item	What it is
`pub const DEFAULT_MAX_ENTRY_BYTES: usize`	Default cap on one entry's serialized size.
`pub const DEFAULT_MAX_BODY_BYTES: usize`	Default cap on a single response body handed to a sink.
`pub const DEFAULT_MAX_REQUEST_BODY_BYTES: usize`	Default cap on a request body.
`pub const DEFAULT_MAX_ENTRIES: usize`	Default cap on entries in one archive.
`pub const DEFAULT_DEADLINE: Duration`	Default wall-clock budget for a whole read.
`pub struct Limits`	Explicit bounds for one read
`pub type Redactor: Box <dyn Fn(& str) -> String + Send + Sync>`	Redaction hook applied inside the reader, before any value reaches a consumer.
`Limits :: fn default() -> Self`	—
`Limits :: fn fmt(& self, f : & mut std::fmt::Formatter <'_>) -> std::fmt::Result`	—
`pub enum HarError`	Everything that can go wrong reading an archive.
`pub struct EntryMeta<'a>`	Position and page context of an entry.
`pub struct HeaderView<'a>`	A name/value header pair.
`pub struct RequestView<'a>`	The request half
`RequestView<'a> :: fn parse_json <T : serde::de::DeserializeOwned>(& self) -> Result <T, serde_json::Error>`	Parse the captured request body as JSON
`RequestView<'a> :: fn form_pairs(& self) -> impl Iterator <Item =(String, String)> + '_`	Iterate `application/x-www-form-urlencoded` pairs, percent-decoded
`RequestView<'a> :: fn form_value(& self, key : & str) -> Option <String>`	First value for `key`, if present.
`pub struct ResponseHead<'a>`	Response status, headers and MIME type — delivered before the body.
`ResponseHead<'_> :: fn is_base64(& self) -> bool`	True when the body is base64-encoded (typically image, font or video bytes).
`ResponseHead<'_> :: fn mime_contains(& self, needle : & str) -> bool`	True when `mime_type` contains `needle` (case-insensitive).
`pub enum ResponseBodyMode`	How a body should be produced, decided by the consumer.
`pub enum EntryAction<S>`	The consumer's decision after seeing the request.
`pub enum HeadDecision`	The consumer's decision after seeing the response head
`pub enum ResponseOutcome`	Why a sink was closed.
`pub trait ResponseSink`	Receives one response
`pub trait EntryVisitor`	Visits entries
`pub struct ReadStats`	What a read observed
`ReadStats :: fn body_ratio(& self) -> f64`	Fraction of entries carrying a body, 0.0–1.0.
`ReadStats :: fn looks_incomplete(& self) -> bool`	True when the capture looks incomplete rather than genuinely empty
`pub struct HarReader`	Reads an archive under `Limits`, driving an `EntryVisitor`.
`HarReader :: fn new() -> Self`	A reader with default limits.
`HarReader :: fn with_limits(limits : Limits) -> Self`	A reader with explicit limits.
`HarReader :: fn read <R : BufRead, V : EntryVisitor>(& self, reader : R, visitor : & mut V,) -> Result <Result <ReadStats, V::Error>, HarError>`	Drive `visitor` over every entry
`HarReader :: fn default() -> Self`	—

`scan`

Item	What it is
`pub trait Scanner`	One shape a `Scanner` looks for.
`pub struct ScanHit`	A recorded hit.
`fn deep_scan <S : Scanner>(value : & Value, scanner : & mut S, max_depth : usize)`	Walk `value` depth-first, driving `scanner`
`pub struct Seen`	Suppresses observations already seen
`Seen :: fn new() -> Self`	An empty tracker.
`Seen :: fn first_time <'a, I : IntoIterator <Item = & 'a str>>(& mut self, parts : I) -> bool`	`true` the first time this combination is offered, `false` on every repeat.
`Seen :: fn len(& self) -> usize`	How many distinct observations have been admitted.
`Seen :: fn is_empty(& self) -> bool`	Whether nothing has been admitted yet.
`pub struct HispanicName`	A name decomposed under Hispanic convention.
`HispanicName :: fn parse(raw : & str) -> Self`	Parse a display name
`HispanicName :: fn shared_line(& self, other : & Self) -> Option <Lineage>`	A proposed lineage link with another name, if their surnames overlap.
`pub enum Lineage`	How two names' surnames overlap.
`pub enum Kin`	A relationship asserted in prose.
`fn kin_lexicon() -> & 'static (& 'static str, Kin)`	Lower-cased phrase → relationship
`fn kin_in(text : & str) -> Vec <Kin>`	Every kinship phrase asserted in `text`, deduplicated.
`pub struct ProposedLink`	A proposed link between two observed people
`pub enum LinkBasis`	Evidence supporting a proposed link.
`fn propose_surname_links(names : & String) -> Vec <ProposedLink>`	Propose surname-based links across observed names
`fn surname_index(names : & String) -> BTreeMap <String, Vec <String>>`	Group observed names by paternal, then maternal surname — the skeleton of a family tree.

Re-exports. Exported here, defined elsewhere.

Export	Defined in
`{deep_scan,ScanHit,Scanner,Seen}`	`scan::{deep_scan,ScanHit,Scanner,Seen}`

Boundary

Depends on no other workspace tier.

Shares tier infrastructure with 82 other crates: infrastructure-acquire, infrastructure-adapters-google-calendar, infrastructure-adapters-google-gmail, infrastructure-adapters-google-places, infrastructure-adapters-google-trends, infrastructure-adapters-shodan, infrastructure-adapters-yelp, infrastructure-agent, … (82 total).

_What this crate deliberately does NOT own is a judgment. No committed registry records one for it, so none is stated here._

Where it sits

Tier (ontology)	`infrastructure`
Architectural role (taxonomy)	`unclassified (baselined)`
Location	`crates/infrastructure/har`
Vocabulary in force (lexicon)	`current`

Dependencies

Runtime, from outside the workspace.

Crate	Requirement	Features	Optional	Only on
`base64`	`^0.22`	—	no	always
`serde`	`^1`	`derive`	no	always
`serde_json`	`^1`	—	no	always
`thiserror`	`^2`	—	no	always

Development, from outside the workspace.

Crate	Requirement	Features	Optional	Only on
`reqwest`	`^0.12`	`blocking`, `json`	no	always
`serde_json`	`^1`	—	no	always

Build. None.

Depended on by. 1 workspace crate.

Signal flow — what reaches this crate, and what it reaches.

flowchart LR
  n_tools_sighting_ingest["tools-sighting-ingest"] -->|uses| SELF
  SELF["infrastructure-har"]
  classDef self fill:#1f883d,stroke:#1f883d,color:#fff;
  class SELF self;

Feature flags

No Cargo features are defined: every capability is unconditional, so no consumer can receive a half-wired crate.

Targets

Kind	Name	Source
example	`push_sniffies`	`examples/push_sniffies.rs`
example	`scan_hars`	`examples/scan_hars.rs`
example	`scan_sniffies`	`examples/scan_sniffies.rs`
example	`survey`	`examples/survey.rs`
lib	`infrastructure_har`	`src/lib.rs`

Error model

Error type	Named by
`HarError`	declared, no public signature returns it

Operational characteristics

Property	Evidence
async public surface	none detected
async runtime	none detected
database access	none detected
network I/O	none detected
unsafe code	none detected
environment variables	none detected

No unsafe block, unsafe fn, unsafe impl or unsafe trait was found by the parser anywhere in this crate's source.

Configuration

No environment variable is read with a literal name anywhere in this crate. A variable whose key is computed at run time cannot be listed here, and is not claimed to be absent.

1 workspace crate depends on this one: tools-sighting-ingest.

Verification

Kind	Count
Unit tests	15
Integration tests	0
Examples	4
Doctests	0

Evidence by module. How often each public module is named by something executable.

Module	Tests	Examples	Consumers
`crate root`	20	12	11
`scan`	13	8	0

What the tests establish, by name:

body_cap_is_an_error_not_a_truncation — src/lib.rs
form_pairs_preserves_duplicate_keys — src/lib.rs
head_decision_abandons_unwanted_mime — src/lib.rs
incomplete_capture_is_detected_not_reported_as_empty — src/lib.rs
percent_decode_is_total_on_binary_input — src/lib.rs
reads_entries_and_counts_bodies — src/lib.rs
redact_applies_to_header_values_before_any_consumer_sees_them — src/lib.rs
deep_scan_bounds_depth_and_reports_paths — src/scan.rs
detects_shared_lines — src/scan.rs
folds_particles_into_the_following_surname — src/scan.rs
kinship_phrases_in_both_languages — src/scan.rs
parses_compound_surnames — src/scan.rs
seen_suppresses_repeats_and_respects_field_boundaries — src/scan.rs
siblings_share_both_surnames — src/scan.rs
surname_index_groups_lines — src/scan.rs

Documentation coverage

Measure	Documented	Total
Public items with rustdoc	49	52
Public modules with a `//!` block	1	1

pie showData
    title Public items with rustdoc
    "Documented" : 49
    "No rustdoc detected" : 3

Metrics

Metric	Value
Rust source files	2
Source lines	1623
Code lines	1185
Public API items	52
Public modules	1
Tests	15
Examples	4
Cargo features	0
Direct runtime dependencies	4
Workspace reverse dependencies	1

pie showData
    title Public API by kind
    "constant" : 5
    "enum" : 8
    "function" : 5
    "method" : 19
    "struct" : 11
    "trait" : 3
    "type alias" : 1

pie showData
    title Rust source composition
    "Code" : 1185
    "Blank or comment" : 438

Generation

Rendered by tools-corpus corpus readme from repository evidence alone, renderer schema 2, lexicon current. No model, network service or database was consulted. Regenerate with tools-corpus corpus readme --write; verify with --check.

What it is for

Capabilities

crate root

EntryAction

EntryMeta

EntryVisitor

HarError

HarReader

HeadDecision

HeaderView

Limits

ReadStats

Redactor

RequestView

RequestView<'a>

ResponseBodyMode

ResponseHead

ResponseHead<'_>

ResponseOutcome

ResponseSink

scan (other)

HispanicName

Kin

Lineage

LinkBasis

ProposedLink

ScanHit

Scanner

Seen

How to use it

Module structure

Public surface

Boundary

Where it sits

Dependencies

Feature flags

Targets

Error model

Operational characteristics

Configuration

Related capabilities

Verification

Documentation coverage

Metrics

Generation

`crate root`

`EntryAction`

`EntryMeta`

`EntryVisitor`

`HarError`

`HarReader`

`HeadDecision`

`HeaderView`

`Limits`

`ReadStats`

`Redactor`

`RequestView`

`RequestView<'a>`

`ResponseBodyMode`

`ResponseHead`

`ResponseHead<'_>`

`ResponseOutcome`

`ResponseSink`

`scan (other)`

`HispanicName`

`Kin`

`Lineage`

`LinkBasis`

`ProposedLink`

`ScanHit`

`Scanner`

`Seen`