infrastructure-fetcher

Content-addressed URL fetcher with sha256 dedup, per-host rate limits, and reqwest→bridge executor fallback

What it is for

infrastructure-fetcher: content-addressed URL fetching with scope-enforced dedup.

Given a URL and routing/validation hints, Fetcher drives an executor stack (reqwest → bridge → …), content-addresses the response by sha256, respects per-host rate limits, and writes the bytes to infrastructure_storage exactly once per (scope, content) pair.

Layer

Infrastructure. Depends on infrastructure_storage and operations_rate_limit.

Key types

• Fetcher — entry point; construct via FetcherBuilder.
• TenantScopedFetcher — tenant-bound handle, dedup scoped per tenant.
• FetchHints — { routing: RoutingHints, validation: ValidationPolicy }.
• FetchOutcome — { stored_file, fetch_status, executor_used, final_url }.

See docs/userstories/sprint-0.1-fetcher.md for the full spec + Gate 1.5 consensus log.

Capabilities

No public items.

How to use it

No examples/ target and no doctest in this crate's rustdoc. The tests listed under Verification are the closest executable usage.

Module structure

No public modules: the crate root is its whole surface.

Public surface

No public items.

Re-exports. Exported here, defined elsewhere.

Boundary

Reaches into operations.

Shares tier infrastructure with 82 other crates: infrastructure-acquire, infrastructure-adapters-google-calendar, infrastructure-adapters-google-gmail, infrastructure-adapters-google-places, infrastructure-adapters-google-trends, infrastructure-adapters-shodan, infrastructure-adapters-yelp, infrastructure-agent, … (82 total).

_What this crate deliberately does NOT own is a judgment. No committed registry records one for it, so none is stated here._

Where it sits

Tier flow. Which tiers this crate's own edges cross.

flowchart LR
  n_infrastructure["infrastructure"] --> n_operations["operations"]

Dependencies

Runtime, in this workspace.

Runtime, from outside the workspace.

Development, from outside the workspace.

Build. None.

Depended on by. 29 workspace crates.

Signal flow — what reaches this crate, and what it reaches.

flowchart LR
  consumers["29 consumer crates"] -->|uses| SELF
  SELF["infrastructure-fetcher"]
  SELF -->|runtime| runtime_deps["2 workspace crates"]
  classDef self fill:#1f883d,stroke:#1f883d,color:#fff;
  class SELF self;

Feature flags

flowchart LR
  n_test_util["test-util"]

Targets

Error model

No public error type was detected: no public item declares a type named *Error, and no public signature returns one.

Operational characteristics

No unsafe block, unsafe fn, unsafe impl or unsafe trait was found by the parser anywhere in this crate's source.

Configuration

No environment variable is read with a literal name anywhere in this crate. A variable whose key is computed at run time cannot be listed here, and is not claimed to be absent.

Related capabilities

29 workspace crates depend on this one: application-fetcher, application-party-discovery, application-wiki, content-assets, infrastructure-acquire, infrastructure-adapters-google-gmail, infrastructure-adapters-google-places, infrastructure-adapters-google-trends, infrastructure-adapters-shodan, infrastructure-adapters-yelp, infrastructure-agent, infrastructure-ai, … (29 total).

Verification

What the tests establish, by name:

• httpbin_round_trip_downloads_and_dedups — tests/httpbin.rs
• egress_default_still_refuses_loopback_literal — tests/safe_http_integration.rs
• internal_client_reaches_allowlisted_loopback_exact_port — tests/safe_http_integration.rs
• internal_client_refuses_non_allowlisted_port_on_listed_host — tests/safe_http_integration.rs
• internal_client_stays_strict_for_non_allowlisted_internal_host — tests/safe_http_integration.rs
• preflight_rejects_metadata_address_and_non_http_scheme — tests/safe_http_integration.rs
• read_body_capped_aborts_when_body_exceeds_cap — tests/safe_http_integration.rs
• read_body_capped_returns_body_within_cap — tests/safe_http_integration.rs
• safe_client_refuses_localhost_via_pinned_resolver — tests/safe_http_integration.rs
• safe_client_refuses_loopback_literal_with_ssrf_error — tests/safe_http_integration.rs
• builder_rejects_bare_build_with_no_general_purpose_executor — src/builder.rs
• builder_rejects_duplicate_executors_with_duplicate_executor — src/builder.rs
• builder_rejects_empty_executor_list_with_no_general_purpose — src/builder.rs
• builder_rejects_missing_rate_limiter_with_rate_limiter_required — src/builder.rs
• builder_rejects_missing_storage_with_storage_required — src/builder.rs
• builder_valid_build_succeeds — src/builder.rs
• builder_with_bridge_and_reqwest_both_register — src/builder.rs
• builder_with_executor_escape_hatch_accepts_custom_impl — src/builder.rs
• builder_with_selector_overrides_default — src/builder.rs
• executor_error_blocked_or_challenged_includes_reason — src/errors.rs
• executor_error_converts_into_fetch_error — src/errors.rs
• executor_error_http_status_includes_code_and_message — src/errors.rs
• executor_error_not_available_names_the_kind — src/errors.rs
• fetch_error_cancelled_displays_cleanly — src/errors.rs
• fetch_error_content_type_mismatch_names_both_sides — src/errors.rs
• fetch_error_rate_limited_display_is_informative — src/errors.rs
• fetch_error_too_large_shows_limit_and_actual — src/errors.rs
• fetch_error_validation_failed_includes_reason — src/errors.rs
• fetcher_build_error_duplicate_executor_names_kind — src/errors.rs
• fetcher_build_error_no_general_purpose_mentions_reqwest_or_bridge — src/errors.rs
• _… 94 more_

Documentation coverage

Metrics

pie showData
    title Rust source composition
    "Code" : 3010
    "Blank or comment" : 1294

Generation

Rendered by tools-corpus corpus readme from repository evidence alone, renderer schema 2, lexicon current. No model, network service or database was consulted. Regenerate with tools-corpus corpus readme --write; verify with --check.