operations-link-graph

Ethical link-following traversal: SSRF-hardened frontier admission (ValidatedUrl), robots.txt compliance, and an async jittered outbound host-pacer composing operations-rate-limit

What it is for

# operations-link-graph

Ethical link-following traversal orchestration: an SSRF-hardened frontier (ValidatedUrl, Frontier), robots.txt compliance (RobotsPolicy), an async jittered outbound host-pacer composing operations-rate-limit (HostPacer), and typed traversal-progress tracking (Traversal, TraversalOutcome).

Lifted from the source's link-following analyzer and robots-policy modules, its batch-orchestration service, and its robots.txt/rate-limiter ethics modules — see CHANGELOG.md for the full provenance (exact paths/types) and deviation notes.

Layer

Operations (see docs/architecture/layers.toml's AC-layer resolution note) — a runtime traversal/pacing orchestration loop, not a data shape. Depends on foundation's foundation-web-analysis (the Analyzer trait), infrastructure's infrastructure-fetcher (SafeHttpClient, is_safe_url, read_body_capped), and operations' operations-rate-limit (RateLimitKey/token-bucket, composed as this crate's own async jittered host-pacer's allow/deny decision — the same-layer exception this needs is recorded in layers.toml, matching operations-mail-sync's precedent).

Gate 1.5 Round 2 (binding design decision)

This crate's frontier is filled from PAGE CONTENT (hrefs parsed out of a fetched page, redirects, sitemap entries), not just one operator-supplied seed URL. docs/userstories/sprint-3.46-plumber-harvest.md's Gate 1.5 Round 2 (unanimous, 3-of-4 seats) requires every discovered URL to be re-validated through infrastructure_fetcher::is_safe_url before frontier admission — implemented as ValidatedUrl, the only type Frontier::push accepts, so an unvalidated URL cannot enter the frontier by construction. See validated_url for the full rationale and tests/ssrf_corpus.rs for the required adversarial test corpus.

Capabilities

No public items.

How to use it

No examples/ target and no doctest in this crate's rustdoc. The tests listed under Verification are the closest executable usage.

Module structure

No public modules: the crate root is its whole surface.

Public surface

No public items.

Re-exports. Exported here, defined elsewhere.

Boundary

Reaches into foundation, infrastructure.

Shares tier operations with 40 other crates: operations-approval-workflow, operations-assessments, operations-block-imaging, operations-boot-media, operations-browser-agent-worker, operations-camera-discovery, operations-camera-liveview, operations-camera-registry, … (40 total).

_What this crate deliberately does NOT own is a judgment. No committed registry records one for it, so none is stated here._

Where it sits

Tier flow. Which tiers this crate's own edges cross.

flowchart LR
  n_operations["operations"] --> n_foundation["foundation"]
  n_operations["operations"] --> n_infrastructure["infrastructure"]

Dependencies

Runtime, in this workspace.

Runtime, from outside the workspace.

Development, from outside the workspace.

Build. None.

Depended on by. Nothing in this workspace.

Signal flow — what reaches this crate, and what it reaches.

flowchart LR
  SELF["operations-link-graph"]
  SELF -->|runtime| n_foundation_web_analysis["foundation-web-analysis"]
  SELF -->|runtime| n_infrastructure_fetcher["infrastructure-fetcher"]
  SELF -->|runtime| n_operations_rate_limit["operations-rate-limit"]
  classDef self fill:#1f883d,stroke:#1f883d,color:#fff;
  class SELF self;

Feature flags

No Cargo features are defined: every capability is unconditional, so no consumer can receive a half-wired crate.

Targets

Error model

No public error type was detected: no public item declares a type named *Error, and no public signature returns one.

Operational characteristics

No unsafe block, unsafe fn, unsafe impl or unsafe trait was found by the parser anywhere in this crate's source.

Configuration

No environment variable is read with a literal name anywhere in this crate. A variable whose key is computed at run time cannot be listed here, and is not claimed to be absent.

Related capabilities

No workspace crate depends on this one.

Verification

What the tests establish, by name:

• dot_segment_relative_resolution_cannot_escape_the_base_origin — tests/ssrf_corpus.rs
• every_discovered_url_in_the_adversarial_corpus_is_rejected — tests/ssrf_corpus.rs
• every_seed_in_the_adversarial_corpus_is_rejected — tests/ssrf_corpus.rs
• legitimate_public_urls_are_still_accepted — tests/ssrf_corpus.rs
• normalize_for_dedup_folds_scheme_and_host_case — src/frontier.rs
• normalize_for_dedup_is_idempotent — src/frontier.rs
• normalize_for_dedup_keeps_the_root_slash — src/frontier.rs
• normalize_for_dedup_preserves_path_case — src/frontier.rs
• normalize_for_dedup_preserves_query_value_case — src/frontier.rs
• normalize_for_dedup_strips_fragment_tracking_params_and_www — src/frontier.rs
• normalize_for_dedup_strips_trailing_slash_when_no_query_survives — src/frontier.rs
• pop_returns_entries_in_fifo_order — src/frontier.rs
• push_adds_a_fresh_entry_and_returns_true — src/frontier.rs
• push_rejects_a_duplicate_by_normalized_key — src/frontier.rs
• default_ethical_matches_the_source_two_to_five_second_bounds — src/pacer.rs
• jittered_delay_extends_past_max_delay_to_honor_a_larger_robots_floor — src/pacer.rs
• jittered_delay_stays_within_configured_bounds_with_no_floor — src/pacer.rs
• wait_for_host_honors_a_robots_delay_floor — src/pacer.rs
• wait_for_host_sleeps_within_the_configured_jitter_bounds — src/pacer.rs
• is_allowed_fails_open_when_no_robots_txt_was_fetched — src/robots.rs
• is_allowed_honors_a_cached_disallow — src/robots.rs
• is_allowed_permits_a_path_outside_the_cached_disallow — src/robots.rs
• parse_robots_txt_detects_blocks_all_bots — src/robots.rs
• parse_robots_txt_extracts_disallow_allow_sitemap_and_delay — src/robots.rs
• parse_robots_txt_ignores_rules_under_a_named_non_wildcard_agent — src/robots.rs
• parse_robots_txt_records_sitemap_regardless_of_agent_block — src/robots.rs
• analyzer_name_and_trait_defaults_are_stable — src/traversal.rs
• extract_hrefs_skips_anchor_javascript_mailto_and_tel_links — src/traversal.rs
• traversal_report_counts_each_outcome_kind — src/traversal.rs
• discovered_resolves_a_relative_href_against_its_base — src/validated_url.rs
• _… 3 more_

Documentation coverage

Metrics

pie showData
    title Rust source composition
    "Code" : 914
    "Blank or comment" : 519

Generation

Rendered by tools-corpus corpus readme from repository evidence alone, renderer schema 2, lexicon current. No model, network service or database was consulted. Regenerate with tools-corpus corpus readme --write; verify with --check.