infrastructure tier

infrastructure-acquire

Acquisition switchboard: one neutral entry point that gets a Target from a URL by the least-abnormal CaptureMethod for the circumstance, driven by a per-site SiteProfile, returning a typed AcquisitionReport of what it got and which rung got it. Ladder engine mirrors infrastructure-fetcher's retryable-descent idiom over pluggable Rung impls (HttpFetch + MetadataSidecar always on; BrowserDom/ScreenshotOcr behind the cdp/ocr features). Foundation only: does not build a link graph, not products, not the temporal index -- see docs/userstories/sprint-3.25-acquire.md.

Acquisition switchboard: one neutral entry point that gets a Target from a URL by the least-abnormal CaptureMethod for the circumstance, driven by a per-site SiteProfile, returning a typed AcquisitionReport of what it got and which rung got it. Ladder engine mirrors infrastructure-fetcher's retryable-descent idiom over pluggable Rung impls (HttpFetch + MetadataSidecar always on; BrowserDom/ScreenshotOcr behind the cdp/ocr features). Foundation only: does not build a link graph, not products, not the temporal index -- see docs/userstories/sprint-3.25-acquire.md.

Tierinfrastructure
Roleunclassified (baselined)
Pathcrates/infrastructure/acquire
Edition2021
Targetsinfrastructure_acquire
Public items46 across 12 modules
Tests66

What it is for

infrastructure-acquire -- the acquisition switchboard (sprint 3.25 foundation; see docs/userstories/sprint-3.25-acquire.md).

One neutral library crate gets a Target from a URL by the least-abnormal CaptureMethod for the circumstance, driven by a per-site SiteProfile, and returns a typed AcquisitionReport of what it got and which rung got it.

# What's in this foundation sprint

pure, fully unit-tested.

trait object, unit-tested with mock rungs.

-- always compiled, composing infrastructure-fetcher/infrastructure-storage and domain-domain-survey.

-- optional, no live browser/tesseract dependency in a default build.

# Non-goals (see the sprint story)

forensic bundle, not API/feed vendor adapters, not PDF/perceptual/OCR word-box capabilities. Those compose this crate in later sprints.

# Sync/async bridge (T3)

infrastructure-browser-automation/infrastructure-browser-cdp and infrastructure-screen-capture are synchronous crates (blocking CDP I/O over a dedicated OS thread, blocking xcap display capture). Their rungs bridge into this crate's async fn attempt via tokio::task::spawn_blocking, matching the workspace's established seam for wrapping a sync library crate from async code -- see rungs::browser and rungs::ocr (feature-gated; landing in Commit 3).

Capabilities

MetadataRequest

Top-level composition: run the ladder, then attach the metadata sidecar

Item
pub struct MetadataRequest<'a>
async fn acquire(url : Url, target : Target, capabilities : & Capabilities, rungs : & (CaptureMethod, & dyn Rung), metadata : Option <MetadataRequest <'_>>, started_at : DateTime <Utc>, finished_at : DateTime <Utc>,) -> AcquisitionReport

ladder (other)

The ladder engine: run enabled CaptureMethods in order, stop at the

Item
async fn run_ladder(url : Url, target : Target, capabilities : & Capabilities, available : & (CaptureMethod, & dyn Rung), started_at : DateTime <Utc>, finished_at : DateTime <Utc>,) -> AcquisitionReport

CaptureMethod

CaptureMethod -- the "how" axis of the switchboard (proposal §2).

Item
pub enum CaptureMethod

Capabilities

SiteProfile / Capabilities -- the per-site capability switchboard

Item
pub struct Capabilities
Capabilities :: fn all_auto() -> Self
Capabilities :: fn is_enabled(& self, method : CaptureMethod) -> bool
Capabilities :: fn resolved_order(& self) -> Vec <CaptureMethod>

MethodOrder

SiteProfile / Capabilities -- the per-site capability switchboard

Item
pub enum MethodOrder

RetentionPolicy

SiteProfile / Capabilities -- the per-site capability switchboard

Item
pub enum RetentionPolicy

SiteProfile

SiteProfile / Capabilities -- the per-site capability switchboard

Item
pub struct SiteProfile

AcquisitionReport

AcquisitionReport -- what the ladder got, and which rung got it.

Item
pub struct AcquisitionReport
AcquisitionReport :: fn is_satisfied(& self) -> bool

ArtifactKind

AcquisitionReport -- what the ladder got, and which rung got it.

Item
pub enum ArtifactKind

Attempt

AcquisitionReport -- what the ladder got, and which rung got it.

Item
pub struct Attempt

AttemptResult

AcquisitionReport -- what the ladder got, and which rung got it.

Item
pub enum AttemptResult

StoredArtifact

AcquisitionReport -- what the ladder got, and which rung got it.

Item
pub struct StoredArtifact
StoredArtifact :: fn new(method : CaptureMethod, kind : ArtifactKind, artifact : ArtifactRef) -> Self

ArtifactRef

Rung -- one capture method's contract with the ladder engine.

Item
pub struct ArtifactRef

Rung

Rung -- one capture method's contract with the ladder engine.

Item
pub trait Rung

RungContext

Rung -- one capture method's contract with the ladder engine.

Item
pub struct RungContext

RungError

Rung -- one capture method's contract with the ladder engine.

Item
pub struct RungError
RungError :: fn new(method : CaptureMethod, err : impl std::fmt::Display) -> Self
RungError :: fn fmt(& self, f : & mut std::fmt::Formatter <'_>) -> std::fmt::Result

RungOutcome

Rung -- one capture method's contract with the ladder engine.

Item
pub struct RungOutcome

BrowserDomRung

BrowserDomRung -- the BrowserDom capture method (feature cdp).

Item
pub struct BrowserDomRung<'a>

BrowserDomRung<'_>

BrowserDomRung -- the BrowserDom capture method (feature cdp).

Item
BrowserDomRung<'_> :: fn method(& self) -> CaptureMethod
BrowserDomRung<'_> :: async fn attempt(& self, ctx : & RungContext) -> Result <RungOutcome, RungError>

BrowserDomRung<'a>

BrowserDomRung -- the BrowserDom capture method (feature cdp).

Item
BrowserDomRung<'a> :: fn new(ws_url : impl Into <String>, storage : & 'a dyn StorageBackend, tenant_id : Uuid, capture_screenshot : bool,) -> Self

HttpFetchRung

HttpFetchRung -- the HttpFetch capture method.

Item
pub struct HttpFetchRung<'a>

HttpFetchRung<'_>

HttpFetchRung -- the HttpFetch capture method.

Item
HttpFetchRung<'_> :: fn method(& self) -> CaptureMethod
HttpFetchRung<'_> :: async fn attempt(& self, ctx : & RungContext) -> Result <RungOutcome, RungError>

HttpFetchRung<'a>

HttpFetchRung -- the HttpFetch capture method.

Item
HttpFetchRung<'a> :: fn new(fetcher : & 'a Fetcher, storage : & 'a dyn StorageBackend, tenant_id : Uuid) -> Self

rungs::metadata (other)

MetadataSidecar -- RDAP domain-registration metadata captured alongside

Item
pub const DEFAULT_MAX_RDAP_BYTES: usize

RdapTransport

MetadataSidecar -- RDAP domain-registration metadata captured alongside

Item
pub trait RdapTransport
async fn capture_metadata(transport : & dyn RdapTransport, rdap_url : & Url,) -> Observed <DomainRegistration>

ReqwestRdapTransport

MetadataSidecar -- RDAP domain-registration metadata captured alongside

Item
pub struct ReqwestRdapTransport
ReqwestRdapTransport :: fn new(max_bytes : usize) -> Result <Self, infrastructure_fetcher::SafeHttpError>
ReqwestRdapTransport :: async fn get(& self, url : & Url) -> Result <(u16, Vec <u8>), String>

ScreenshotOcrRung

ScreenshotOcrRung -- the ScreenshotOcr capture method (feature ocr).

Item
pub struct ScreenshotOcrRung<'a, E : OcrEngine>

ScreenshotOcrRung<'_,E>

ScreenshotOcrRung -- the ScreenshotOcr capture method (feature ocr).

Item
ScreenshotOcrRung<'_,E> :: fn method(& self) -> CaptureMethod
ScreenshotOcrRung<'_,E> :: async fn attempt(& self, _ctx : & RungContext) -> Result <RungOutcome, RungError>

ScreenshotOcrRung<'a,E>

ScreenshotOcrRung -- the ScreenshotOcr capture method (feature ocr).

Item
ScreenshotOcrRung<'a,E> :: fn new(ocr : E, storage : & 'a dyn StorageBackend, tenant_id : Uuid) -> Self

Target

Target -- the "what" axis of the switchboard (proposal §2) and its

Item
pub enum Target
Target :: fn is_satisfied(& self, content : & str) -> bool
Target :: fn matched_text(& self, content : & str) -> Option <String>

How to use it

No examples/ target and no doctest in this crate's rustdoc. The tests listed under Verification are the closest executable usage.

Module structure

infrastructure_acquire

flowchart TD
  n_infrastructure_acquire["infrastructure_acquire"]
  n_infrastructure_acquire --> n_acquire["acquire"]
  n_infrastructure_acquire --> n_ladder["ladder"]
  n_infrastructure_acquire --> n_method["method"]
  n_infrastructure_acquire --> n_profile["profile"]
  n_infrastructure_acquire --> n_report["report"]
  n_infrastructure_acquire --> n_rung["rung"]
  n_infrastructure_acquire --> n_rungs["rungs"]
  n_rungs --> n_rungs__browser["browser"]
  n_rungs --> n_rungs__http["http"]
  n_rungs --> n_rungs__metadata["metadata"]
  n_rungs --> n_rungs__ocr["ocr"]
  n_infrastructure_acquire --> n_target["target"]

Public surface

`acquire`

ItemWhat it is
pub struct MetadataRequest<'a>What's needed to run MetadataSidecar alongside the ladder
async fn acquire(url : Url, target : Target, capabilities : & Capabilities, rungs : & (CaptureMethod, & dyn Rung), metadata : Option <MetadataRequest <'_>>, started_at : DateTime <Utc>, finished_at : DateTime <Utc>,) -> AcquisitionReportRun the acquisition ladder against url/target, then -- ONLY if MetadataSidecar is enabled in capabilities AND metadata was supplied -- attach the RDAP sidecar result

`ladder`

ItemWhat it is
async fn run_ladder(url : Url, target : Target, capabilities : & Capabilities, available : & (CaptureMethod, & dyn Rung), started_at : DateTime <Utc>, finished_at : DateTime <Utc>,) -> AcquisitionReportRun the ladder against available rungs, in the order capabilities.resolved_order() resolves to

`method`

ItemWhat it is
pub enum CaptureMethodHow a rung captures a target, ordered least- to most-abnormal per the capture ladder (docs/proposals/web-acquisition-platform.md §3)

`profile`

ItemWhat it is
pub enum MethodOrderWhich order enabled CaptureMethods run in.
pub struct CapabilitiesPer-capability toggles + ordering for one site/profile.
Capabilities :: fn all_auto() -> SelfAll methods enabled, Auto order.
Capabilities :: fn is_enabled(& self, method : CaptureMethod) -> bool
Capabilities :: fn resolved_order(& self) -> Vec <CaptureMethod>The ladder-eligible methods (excludes MetadataSidecar, which is never ladder-ordered), enabled and in effective order
pub enum RetentionPolicyRecorded-not-enforced in v1 (sprint story, Scope IN): a value carried on the report / profile for a downstream retention-enforcement sprint to consume, never acted on here.
pub struct SiteProfileOne site's switchboard settings.

`report`

ItemWhat it is
pub enum AttemptResultOne rung's disposition within a ladder run.
pub struct AttemptOne rung's attempt record, in ladder order.
pub enum ArtifactKindWhich stored artifact a StoredArtifact is.
pub struct StoredArtifactA stored-artifact identity produced during acquisition.
StoredArtifact :: fn new(method : CaptureMethod, kind : ArtifactKind, artifact : ArtifactRef) -> Self
pub struct AcquisitionReportWhat one acquisition run did: which rung (if any) satisfied the target, every attempt in ladder order with its rejection/error reason, every stored-artifact identity produced along the way, and the metadata sidecar result (if the capability was on) -- captured alongside content, never counted as target satisfaction.
AcquisitionReport :: fn is_satisfied(& self) -> bool

`rung`

ItemWhat it is
pub struct RungContextWhat one rung attempt was asked to do, and when
pub struct ArtifactRefA stored-artifact identity a rung produced (content bytes, a screenshot, ...)
pub struct RungOutcomeWhat a rung attempt produced, before target evaluation.
pub struct RungErrorA rung's typed failure
RungError :: fn new(method : CaptureMethod, err : impl std::fmt::Display) -> Self
RungError :: fn fmt(& self, f : & mut std::fmt::Formatter <'_>) -> std::fmt::Result
pub trait RungOne capture method's contract with the ladder engine

`rungs::browser`

ItemWhat it is
pub struct BrowserDomRung<'a>BrowserDom rung: connect to a running Chrome/Chromium over CDP, navigate to ctx.url, read the settled DOM as HTML, and -- when capture_screenshot is on -- store a PNG screenshot as a retained, first-class artifact INDEPENDENT of whether ScreenshotOcr ever runs (sprint story, Scope IN)
BrowserDomRung<'a> :: fn new(ws_url : impl Into <String>, storage : & 'a dyn StorageBackend, tenant_id : Uuid, capture_screenshot : bool,) -> Self
BrowserDomRung<'_> :: fn method(& self) -> CaptureMethod
BrowserDomRung<'_> :: async fn attempt(& self, ctx : & RungContext) -> Result <RungOutcome, RungError>

`rungs::http`

ItemWhat it is
pub struct HttpFetchRung<'a>HttpFetch rung: a infrastructure-fetcher GET, decoded lossily to text for target evaluation, content-addressed via the same StorageBackend the Fetcher was built with
HttpFetchRung<'a> :: fn new(fetcher : & 'a Fetcher, storage : & 'a dyn StorageBackend, tenant_id : Uuid) -> Self
HttpFetchRung<'_> :: fn method(& self) -> CaptureMethod
HttpFetchRung<'_> :: async fn attempt(& self, ctx : & RungContext) -> Result <RungOutcome, RungError>

`rungs::metadata`

ItemWhat it is
pub trait RdapTransportThe sidecar's HTTP seam -- lets tests supply canned (status, body) pairs with no live RDAP server, and is what a caller swaps for a fake in unit tests.
pub const DEFAULT_MAX_RDAP_BYTES: usizeDefault cap on an RDAP response body -- generous for RDAP's small JSON payloads, small enough to bound a hostile/misconfigured server.
pub struct ReqwestRdapTransportThe real transport: a minimal capped GET, built on SafeHttpClient::inner() so the pinned DNS resolver protects domain hosts (see the module doc's SSRF section)
ReqwestRdapTransport :: fn new(max_bytes : usize) -> Result <Self, infrastructure_fetcher::SafeHttpError># Errors Propagates SafeHttpClient::builder().build()'s error (TLS backend init failure) -- essentially infallible in practice.
ReqwestRdapTransport :: async fn get(& self, url : & Url) -> Result <(u16, Vec <u8>), String>
async fn capture_metadata(transport : & dyn RdapTransport, rdap_url : & Url,) -> Observed <DomainRegistration>Capture the RDAP-based metadata sidecar for rdap_url

`rungs::ocr`

ItemWhat it is
pub struct ScreenshotOcrRung<'a, E : OcrEngine>ScreenshotOcr rung: whole-monitor screenshot (infrastructure-screen-capture, sync) -> local OCR text (infrastructure-ocr-tesseract, async, on-host, no egress)
ScreenshotOcrRung<'a,E> :: fn new(ocr : E, storage : & 'a dyn StorageBackend, tenant_id : Uuid) -> Self
ScreenshotOcrRung<'_,E> :: fn method(& self) -> CaptureMethod
ScreenshotOcrRung<'_,E> :: async fn attempt(& self, _ctx : & RungContext) -> Result <RungOutcome, RungError>

`target`

ItemWhat it is
pub enum TargetWhat to get from a page
Target :: fn is_satisfied(& self, content : & str) -> boolEvaluate the sufficiency predicate against a rung's raw output text (HTML for HttpFetch/BrowserDom; plain text for ScreenshotOcr, which trivially never satisfies Selector/Field since OCR output carries no markup -- only Everything/LinkText-as-substring can pass against it)
Target :: fn matched_text(& self, content : & str) -> Option <String>Evaluate the predicate AND return the target-relevant text it resolved to when satisfied -- the ladder attaches this to a Satisfied attempt (sprint AC: "the extracted target text")

Re-exports. Exported here, defined elsewhere.

ExportDefined in
BrowserDomRungrungs::browser::BrowserDomRung
CaptureMethodmethod::CaptureMethod
HttpFetchRungrungs::http::HttpFetchRung
ScreenshotOcrRungrungs::ocr::ScreenshotOcrRung
Targettarget::Target
run_ladderladder::run_ladder
{AcquisitionReport,ArtifactKind,Attempt,AttemptResult,StoredArtifact}report::{AcquisitionReport,ArtifactKind,Attempt,AttemptResult,StoredArtifact}
{ArtifactRef,Rung,RungContext,RungError,RungOutcome}rung::{ArtifactRef,Rung,RungContext,RungError,RungOutcome}
{Capabilities,MethodOrder,RetentionPolicy,SiteProfile}profile::{Capabilities,MethodOrder,RetentionPolicy,SiteProfile}
{acquireasrun_acquire,MetadataRequest}acquire::{acquireasrun_acquire,MetadataRequest}
{capture_metadata,RdapTransport,ReqwestRdapTransport}rungs::metadata::{capture_metadata,RdapTransport,ReqwestRdapTransport}

Boundary

Reaches into domain.

Shares tier infrastructure with 82 other crates: infrastructure-adapters-google-calendar, infrastructure-adapters-google-gmail, infrastructure-adapters-google-places, infrastructure-adapters-google-trends, infrastructure-adapters-shodan, infrastructure-adapters-yelp, infrastructure-agent, infrastructure-ai, … (82 total).

_What this crate deliberately does NOT own is a judgment. No committed registry records one for it, so none is stated here._

Where it sits

Tier (ontology)infrastructure
Architectural role (taxonomy)unclassified (baselined)
Locationcrates/infrastructure/acquire
Vocabulary in force (lexicon)current

Tier flow. Which tiers this crate's own edges cross.

flowchart LR
  n_infrastructure["infrastructure"] --> n_domain["domain"]

Dependencies

Runtime, in this workspace.

CrateTierOptionalOnly on
`domain-domain-survey`domainnoalways
`infrastructure-browser-automation`infrastructureyesalways
`infrastructure-browser-cdp`infrastructureyesalways
`infrastructure-fetcher`infrastructurenoalways
`infrastructure-html-text`infrastructurenoalways
`infrastructure-ocr-engine`infrastructureyesalways
`infrastructure-ocr-tesseract`infrastructureyesalways
`infrastructure-screen-capture`infrastructureyesalways
`infrastructure-storage`infrastructurenoalways

Runtime, from outside the workspace.

CrateRequirementFeaturesOptionalOnly on
async-trait^0.1noalways
base64^0.22yesalways
chrono^0.4serdenoalways
lol_html^3.0.1noalways
reqwest^0.12rustls-tlsnoalways
serde^1derive, derivenoalways
serde_json^1noalways
thiserror^2noalways
tokio^1fullnoalways
url^2noalways
uuid^1v4, v7, serde, js, v4, serdenoalways

Development, in this workspace.

CrateTierOptionalOnly on
`infrastructure-fetcher`infrastructurenoalways
`operations-rate-limit`operationsnoalways

Development, from outside the workspace.

CrateRequirementFeaturesOptionalOnly on
tempfile^3noalways
wiremock^0.6noalways

Build. None.

Depended on by. Nothing in this workspace.

Signal flow — what reaches this crate, and what it reaches.

flowchart LR
  SELF["infrastructure-acquire"]
  SELF -->|development| n_infrastructure_fetcher["infrastructure-fetcher"]
  SELF -->|development| n_operations_rate_limit["operations-rate-limit"]
  SELF -->|runtime| n_domain_domain_survey["domain-domain-survey"]
  SELF -->|runtime| n_infrastructure_browser_automation["infrastructure-browser-automation"]
  SELF -->|runtime| n_infrastructure_browser_cdp["infrastructure-browser-cdp"]
  SELF -->|runtime| n_infrastructure_fetcher["infrastructure-fetcher"]
  SELF -->|runtime| n_infrastructure_html_text["infrastructure-html-text"]
  SELF -->|runtime| n_infrastructure_ocr_engine["infrastructure-ocr-engine"]
  SELF -->|runtime| n_infrastructure_ocr_tesseract["infrastructure-ocr-tesseract"]
  SELF -->|runtime| n_infrastructure_screen_capture["infrastructure-screen-capture"]
  SELF -->|runtime| n_infrastructure_storage["infrastructure-storage"]
  classDef self fill:#1f883d,stroke:#1f883d,color:#fff;
  class SELF self;

Feature flags

FeatureEnablesOn by default
cdpdep:infrastructure-browser-automation, dep:infrastructure-browser-cdp, dep:base64no
defaultyes
ocrdep:infrastructure-screen-capture, dep:infrastructure-ocr-tesseract, dep:infrastructure-ocr-engineno
flowchart LR
  n_cdp["cdp"] --> n_dep_infrastructure_browser_automation["dep:infrastructure-browser-automation"]
  n_cdp["cdp"] --> n_dep_infrastructure_browser_cdp["dep:infrastructure-browser-cdp"]
  n_cdp["cdp"] --> n_dep_base64["dep:base64"]
  n_default["default"]
  n_ocr["ocr"] --> n_dep_infrastructure_screen_capture["dep:infrastructure-screen-capture"]
  n_ocr["ocr"] --> n_dep_infrastructure_ocr_tesseract["dep:infrastructure-ocr-tesseract"]
  n_ocr["ocr"] --> n_dep_infrastructure_ocr_engine["dep:infrastructure-ocr-engine"]

Targets

KindNameSource
libinfrastructure_acquire`src/lib.rs`

Error model

Error typeNamed by
RungErrordeclared, no public signature returns it

Operational characteristics

PropertyEvidence
async public surfaceyes
async runtimeyes
database accessnone detected
network I/Oyes
unsafe codenone detected
environment variablesyes

No unsafe block, unsafe fn, unsafe impl or unsafe trait was found by the parser anywhere in this crate's source.

Configuration

VariableRead in
ACQUIRE_TEST_CDP_WS_URLsrc/rungs/browser.rs

No workspace crate depends on this one.

Verification

KindCount
Unit tests66
Integration tests0
Examples0
Doctests0

Evidence by module. How often each public module is named by something executable.

ModuleTestsExamplesConsumers
acquire200
ladder100
method100
profile400
report500
rung500
rungs::browser100
rungs::http100
rungs::metadata400
rungs::ocr100
target100

What the tests establish, by name:

Documentation coverage

MeasureDocumentedTotal
Public items with rustdoc3146
Public modules with a //! block1212
pie showData
    title Public items with rustdoc
    "Documented" : 31
    "No rustdoc detected" : 15

Metrics

MetricValue
Rust source files13
Source lines2527
Code lines1809
Public API items46
Public modules12
Tests66
Examples0
Cargo features3
Direct runtime dependencies20
Workspace reverse dependencies0
pie showData
    title Public API by kind
    "constant" : 1
    "enum" : 6
    "function" : 3
    "method" : 20
    "struct" : 14
    "trait" : 2
pie showData
    title Rust source composition
    "Code" : 1809
    "Blank or comment" : 718

Generation

Rendered by tools-corpus corpus readme from repository evidence alone, renderer schema 2, lexicon current. No model, network service or database was consulted. Regenerate with tools-corpus corpus readme --write; verify with --check.

All infrastructure · Manual