domain-domain-survey

Non-contact domain survey collectors: RDAP-first registration lookup, per-source rate limiting, and preserve-before-parse capture. Every result is a three-state observation.

Tier	domain
Role	unclassified (baselined)
Path	`crates/domain/domain-survey`
Edition	2021
Targets	domain_domain_survey
Public items	39 across 5 modules
Tests	38

What it is for

Non-contact domain survey.

Collectors that establish what is published about a domain — who registered it, when, with which nameservers and mail configuration, what certificates exist, and whether a website ever stood there.

# Passive means we do not ask the subject

Every collector here consults a third party that already holds the answer: a registry, a certificate transparency log, a web archive, a public recursive resolver. The subject's own systems are never contacted, so the subject learns nothing — not because anything is concealed, but because nothing is requested from them.

That is a property of the sources, not of our cleverness, and it is why these collectors are DisclosureClass::Quiet rather than requiring authorization.

DisclosureClass::Quiet: domain_research_intel::DisclosureClass::Quiet

# Three rules this crate is built on

1. Preserve before parse. Bytes are hashed and kept before anything interprets them, including bytes that failed — see preserve. 2. Rate limit per source, and fail closed. An unconfigured source is denied. Overrunning a registry does not merely slow us down, it manufactures undetermined results — see rate_limit. 3. Never record an undetermined result as a negative finding. Every collector returns Observed<T>, and an absence requires a witness that cannot be minted from a failure.

Observed<T>: domain_research_intel::Observed

# Why rule 3 has a crate built around it

An ad-hoc script once recorded HTTP 502 from a certificate-transparency API as zero certificates ever issued, and that reading — combined with "no archive history" — made a legitimate eight-year-old company look like a purpose-built fraud instrument. It had 301 certificates. The API was down.

The script had no three-state discipline, no rate limiting, and preserved nothing. This crate exists so that collection does not happen that way again.

Capabilities

`RawCapture`

Preserve before parse.

Item
`pub struct RawCapture`
`RawCapture :: fn preserve(source : impl Into <String>, request : impl Into <String>, captured_at : DateTime <Utc>, http_status : Option <u16>, disclosure : DisclosureClass, body : & u8,) -> Self`
`RawCapture :: fn parsed_by(mut self, parser_version : impl Into <String>) -> Self`
`RawCapture :: fn matches(& self, body : & u8) -> bool`
`RawCapture :: fn is_transport_success(& self) -> bool`
`RawCapture :: fn is_parsed(& self) -> bool`

`Backoff`

Per-source rate limiting.

Item
`pub struct Backoff`
`Backoff :: fn default() -> Self`
`Backoff :: const fn new(base_ms : i64, max_ms : i64, max_attempts : u8) -> Self`
`Backoff :: fn delay_before(& self, attempt : u8) -> Option <Duration>`
`Backoff :: const fn max_attempts(& self) -> u8`

`Permit`

Per-source rate limiting.

Item
`pub enum Permit`
`Permit :: const fn is_go(self) -> bool`
`Permit :: const fn delay(self) -> Option <Duration>`

`RateLimiter`

Per-source rate limiting.

Item
`pub struct RateLimiter`
`RateLimiter :: fn new() -> Self`
`RateLimiter :: fn with_defaults() -> Self`
`RateLimiter :: fn configure(& mut self, source : impl Into <String>, min_interval_ms : i64)`
`RateLimiter :: fn check(& self, source : & str, now : DateTime <Utc>) -> Permit`
`RateLimiter :: fn record(& mut self, source : & str, now : DateTime <Utc>)`
`RateLimiter :: fn is_configured(& self, source : & str) -> bool`

`rate_limit::defaults (other)`

_No module-level documentation is present in the source._

Item
`pub const VERISIGN_WHOIS_MS: i64`
`pub const REGISTRY_MX_MS: i64`
`pub const REGISTRO_BR_MS: i64`
`pub const CCTLD_DEFAULT_MS: i64`
`pub const RDAP_MS: i64`
`pub const CRT_SH_MS: i64`
`pub const ARCHIVE_CDX_MS: i64`
`pub const PUBLIC_RESOLVER_MS: i64`

`rdap (other)`

RDAP — registration data, parsed into a three-state observation.

Item
`pub const PARSER_VERSION: & str`
`pub const SOURCE: & str`

`DomainRegistration`

RDAP — registration data, parsed into a three-state observation.

Item
`pub struct DomainRegistration`
`DomainRegistration :: fn age_days(& self, now : DateTime <Utc>) -> Option <i64>`
`DomainRegistration :: fn days_since_change(& self, now : DateTime <Utc>) -> Option <i64>`
`fn parse(source : & str, http_status : u16, body : & u8) -> Observed <DomainRegistration>`

`ObservationRow`

Survey store — where a collection run stops being a conversation.

Item
`pub struct ObservationRow`
`ObservationRow :: fn from_observed <T : Serialize>(run_id : impl Into <String>, subject : impl Into <String>, capture : & RawCapture, observed : & Observed <T>,) -> Result <Self, StoreError>`

`StoreError`

Survey store — where a collection run stops being a conversation.

Item
`pub enum StoreError`

`SurveyStore`

Survey store — where a collection run stops being a conversation.

Item
`pub trait SurveyStore`

How to use it

No examples/ target and no doctest in this crate's rustdoc. The tests listed under Verification are the closest executable usage.

Module structure

domain_domain_survey

preserve
rate_limit
rate_limit::defaults
rdap
store

flowchart TD
  n_domain_domain_survey["domain_domain_survey"]
  n_domain_domain_survey --> n_preserve["preserve"]
  n_domain_domain_survey --> n_rate_limit["rate_limit"]
  n_rate_limit --> n_rate_limit__defaults["defaults"]
  n_domain_domain_survey --> n_rdap["rdap"]
  n_domain_domain_survey --> n_store["store"]

Public surface

`preserve`

Item	What it is
`pub struct RawCapture`	A preserved raw response, hashed before any interpretation
`RawCapture :: fn preserve(source : impl Into <String>, request : impl Into <String>, captured_at : DateTime <Utc>, http_status : Option <u16>, disclosure : DisclosureClass, body : & u8,) -> Self`	Preserve bytes
`RawCapture :: fn parsed_by(mut self, parser_version : impl Into <String>) -> Self`	Record which parser interpreted these bytes.
`RawCapture :: fn matches(& self, body : & u8) -> bool`	Whether `body` is the material this capture preserved
`RawCapture :: fn is_transport_success(& self) -> bool`	Whether the transport reported success
`RawCapture :: fn is_parsed(& self) -> bool`	Whether these bytes have been interpreted yet.

`rate_limit`

Item	What it is
`pub enum Permit`	The outcome of asking to make a request.
`Permit :: const fn is_go(self) -> bool`	Whether the request may proceed immediately.
`Permit :: const fn delay(self) -> Option <Duration>`	How long to wait, if waiting is required.
`pub struct RateLimiter`	Per-source rate limiter
`RateLimiter :: fn new() -> Self`	An empty limiter
`RateLimiter :: fn with_defaults() -> Self`	A limiter preconfigured with `defaults` for every source this crate uses.
`RateLimiter :: fn configure(& mut self, source : impl Into <String>, min_interval_ms : i64)`	Set the minimum interval between requests to `source`.
`RateLimiter :: fn check(& self, source : & str, now : DateTime <Utc>) -> Permit`	May we call `source` at `now`? An unconfigured source is denied, not permitted
`RateLimiter :: fn record(& mut self, source : & str, now : DateTime <Utc>)`	Record that a request to `source` was made at `now`
`RateLimiter :: fn is_configured(& self, source : & str) -> bool`	Whether `source` has a configured limit.
`pub struct Backoff`	Backoff schedule for retrying an undetermined result
`Backoff :: fn default() -> Self`	—
`Backoff :: const fn new(base_ms : i64, max_ms : i64, max_attempts : u8) -> Self`	A custom schedule.
`Backoff :: fn delay_before(& self, attempt : u8) -> Option <Duration>`	How long to wait before attempt number `attempt` (1-based), or `None` when the budget is spent.
`Backoff :: const fn max_attempts(& self) -> u8`	The attempt budget.

`rate_limit::defaults`

Item	What it is
`pub const VERISIGN_WHOIS_MS: i64`	Verisign WHOIS (`.com`/`.net`) — bans by IP for hours.
`pub const REGISTRY_MX_MS: i64`	`registry.mx` (`.mx`/`.com.mx`) — few free queries before refusal.
`pub const REGISTRO_BR_MS: i64`	`registro.br` (`.br`/`.com.br`) — documented published limit, strict.
`pub const CCTLD_DEFAULT_MS: i64`	Any other ccTLD registry — assume strict until measured.
`pub const RDAP_MS: i64`	RDAP endpoints — generally friendlier than port-43 WHOIS.
`pub const CRT_SH_MS: i64`	`crt.sh` — frequently 502s under load.
`pub const ARCHIVE_CDX_MS: i64`	Internet Archive CDX — 429s under load.
`pub const PUBLIC_RESOLVER_MS: i64`	Public recursive resolver — effectively free.

`rdap`

Item	What it is
`pub const PARSER_VERSION: & str`	Parser identity recorded on every capture, so a reparse is attributable.
`pub const SOURCE: & str`	The rate-limit source key
`pub struct DomainRegistration`	What a registry says about a domain
`DomainRegistration :: fn age_days(& self, now : DateTime <Utc>) -> Option <i64>`	Age in whole days at `now`, when a registration date was returned.
`DomainRegistration :: fn days_since_change(& self, now : DateTime <Utc>) -> Option <i64>`	Days since the record last changed, when the registry returned one.
`fn parse(source : & str, http_status : u16, body : & u8) -> Observed <DomainRegistration>`	Parse an RDAP response into an observation

`store`

Item	What it is
`pub struct ObservationRow`	One recorded observation about one subject from one source
`ObservationRow :: fn from_observed <T : Serialize>(run_id : impl Into <String>, subject : impl Into <String>, capture : & RawCapture, observed : & Observed <T>,) -> Result <Self, StoreError>`	Build a row from an observation, keeping the three states distinct
`pub enum StoreError`	What a store can refuse to do.
`pub trait SurveyStore`	Persistence for a survey run

Re-exports. Exported here, defined elsewhere.

Export	Defined in
`DomainRegistration`	`rdap::DomainRegistration`
`sha256_hex`	`foundation_audit_log::hash::sha256_hex`
`{AdmiraltyGrade,Confidence,DisclosureClass,EmptySuccess,InfoCredibility,Observed,ObservedExt,Presence,SourceReliability,Unknown,}`	`domain_research_intel::{AdmiraltyGrade,Confidence,DisclosureClass,EmptySuccess,InfoCredibility,Observed,ObservedExt,Presence,SourceReliability,Unknown,}`
`{Backoff,Permit,RateLimiter}`	`rate_limit::{Backoff,Permit,RateLimiter}`
`{ObservationRow,StoreError,SurveyStore}`	`store::{ObservationRow,StoreError,SurveyStore}`
`{sha256_hex,RawCapture}`	`preserve::{sha256_hex,RawCapture}`

Boundary

Reaches into foundation.

Shares tier domain with 41 other crates: domain-agreements, domain-ai-report, domain-billing, domain-catalog, domain-classify, domain-comments, domain-competitive-intel, domain-contact, … (41 total).

_What this crate deliberately does NOT own is a judgment. No committed registry records one for it, so none is stated here._

Where it sits

Tier (ontology)	`domain`
Architectural role (taxonomy)	`unclassified (baselined)`
Location	`crates/domain/domain-survey`
Vocabulary in force (lexicon)	`current`

Tier flow. Which tiers this crate's own edges cross.

flowchart LR
  n_domain["domain"] --> n_foundation["foundation"]

Dependencies

Runtime, in this workspace.

Crate	Tier	Optional	Only on
`domain-research`	domain	no	always
`domain-research-intel`	domain	no	always
`foundation-audit-log`	foundation	no	always

Runtime, from outside the workspace.

Crate	Requirement	Features	Optional	Only on
`chrono`	`^0.4`	`serde`	no	always
`serde`	`^1`	`derive`	no	always
`serde_json`	`^1`	—	no	always
`thiserror`	`^2`	—	no	always
`uuid`	`^1`	`v4`, `v7`, `serde`, `js`	no	always

Development, from outside the workspace.

Crate	Requirement	Features	Optional	Only on
`tokio-test`	`^0.4`	—	no	always

Build. None.

Depended on by. 2 workspace crates.

Signal flow — what reaches this crate, and what it reaches.

flowchart LR
  n_application_domain_survey["application-domain-survey"] -->|uses| SELF
  n_infrastructure_acquire["infrastructure-acquire"] -->|uses| SELF
  SELF["domain-domain-survey"]
  SELF -->|runtime| n_domain_research["domain-research"]
  SELF -->|runtime| n_domain_research_intel["domain-research-intel"]
  SELF -->|runtime| n_foundation_audit_log["foundation-audit-log"]
  classDef self fill:#1f883d,stroke:#1f883d,color:#fff;
  class SELF self;

Feature flags

No Cargo features are defined: every capability is unconditional, so no consumer can receive a half-wired crate.

Targets

Kind	Name	Source
lib	`domain_domain_survey`	`src/lib.rs`

Error model

Error type	Named by
`StoreError`	declared, no public signature returns it

Operational characteristics

Property	Evidence
async public surface	none detected
async runtime	none detected
database access	none detected
network I/O	none detected
unsafe code	none detected
environment variables	none detected

No unsafe block, unsafe fn, unsafe impl or unsafe trait was found by the parser anywhere in this crate's source.

Configuration

No environment variable is read with a literal name anywhere in this crate. A variable whose key is computed at run time cannot be listed here, and is not claimed to be absent.

2 workspace crates depend on this one: application-domain-survey, infrastructure-acquire.

Verification

Kind	Count
Unit tests	38
Integration tests	0
Examples	0
Doctests	0

Evidence by module. How often each public module is named by something executable.

Module	Tests	Consumers
`preserve`	1	0
`rate_limit`	3	1
`rate_limit::defaults`	8	0
`rdap`	4	3
`store`	3	0

What the tests establish, by name:

a_genuine_empty_result_is_a_finding — src/lib.rs
passive_collectors_need_no_authorization — src/lib.rs
the_incident_end_to_end — src/lib.rs
a_failed_response_is_still_preserved — src/preserve.rs
capture_round_trips_through_serde — src/preserve.rs
capture_starts_unparsed_then_records_its_parser — src/preserve.rs
digest_detects_altered_bytes — src/preserve.rs
digest_detects_truncation — src/preserve.rs
disclosure_class_travels_with_the_capture — src/preserve.rs
empty_body_is_a_real_capture_not_a_missing_one — src/preserve.rs
html_served_as_200_is_a_transport_success_and_still_unparseable — src/preserve.rs
identical_bodies_hash_identically — src/preserve.rs
sha256_hex_matches_known_vector — src/preserve.rs
backoff_budget_is_finite — src/rate_limit.rs
backoff_first_attempt_is_immediate — src/rate_limit.rs
backoff_grows_exponentially_then_caps — src/rate_limit.rs
backoff_rejects_attempt_zero — src/rate_limit.rs
failed_attempts_still_consume_the_interval — src/rate_limit.rs
first_request_to_a_configured_source_proceeds — src/rate_limit.rs
registries_are_slower_than_resolvers — src/rate_limit.rs
request_after_the_interval_proceeds — src/rate_limit.rs
second_request_inside_the_interval_must_wait — src/rate_limit.rs
sources_are_limited_independently — src/rate_limit.rs
unconfigured_source_fails_closed — src/rate_limit.rs
a_502_can_never_become_absent — src/rdap.rs
a_missing_registration_event_stays_none_rather_than_zero — src/rdap.rs
a_proxy_404_page_is_not_a_registry_saying_unregistered — src/rdap.rs
a_redirect_is_not_an_answer — src/rdap.rs
a_well_formed_rdap_404_is_a_determined_absence — src/rdap.rs
age_is_derived_never_stored — src/rdap.rs
_… 8 more_

Documentation coverage

Measure	Documented	Total
Public items with rustdoc	38	39
Public modules with a `//!` block	4	5

pie showData
    title Public items with rustdoc
    "Documented" : 38
    "No rustdoc detected" : 1

Metrics

Metric	Value
Rust source files	5
Source lines	1333
Code lines	844
Public API items	39
Public modules	5
Tests	38
Examples	0
Cargo features	0
Direct runtime dependencies	8
Workspace reverse dependencies	2

pie showData
    title Public API by kind
    "constant" : 10
    "enum" : 2
    "function" : 1
    "method" : 20
    "struct" : 5
    "trait" : 1

pie showData
    title Rust source composition
    "Code" : 844
    "Blank or comment" : 489

Generation

Rendered by tools-corpus corpus readme from repository evidence alone, renderer schema 2, lexicon current. No model, network service or database was consulted. Regenerate with tools-corpus corpus readme --write; verify with --check.

What it is for

Capabilities

RawCapture

Backoff

Permit

RateLimiter

rate_limit::defaults (other)

rdap (other)

DomainRegistration

ObservationRow

StoreError

SurveyStore

How to use it

Module structure

Public surface

Boundary

Where it sits

Dependencies

Feature flags

Targets

Error model

Operational characteristics

Configuration

Related capabilities

Verification

Documentation coverage

Metrics

Generation

`RawCapture`

`Backoff`

`Permit`

`RateLimiter`

`rate_limit::defaults (other)`

`rdap (other)`

`DomainRegistration`

`ObservationRow`

`StoreError`

`SurveyStore`