infrastructure-embeddings

Provider-agnostic text vectorization (embeddings) for semantic search + RAG: an Embedder trait, a provenance-preserving chunker, a deterministic dev embedder, and (feature-gated) an in-process ONNX BGE-m3 vectorizer.

What it is for

infrastructure-embeddings — provider-agnostic text vectorization for semantic search + RAG.

The surface is deliberately small: one Embedder trait, a provenance- preserving chunk_text splitter, and a dependency-free DeterministicEmbedder so the full chunk → embed → store → search pipeline builds and tests with zero native deps. The real in-process ONNX BGE-m3 impl lives behind the ort feature (OrtBgeM3Embedder) and drops in behind the same trait.

Design + rationale: ADR 0016. Consumers own the vector store (pgvector per tenant) and any privilege/isolation policy — this crate never performs external network egress on the default path, so privileged text stays in-process.

use infrastructure_embeddings::{chunk_text, ChunkOptions, DeterministicEmbedder, Embedder};
# async fn demo() -> Result<(), infrastructure_embeddings::EmbeddingError> {
let chunks = chunk_text("some long document text...", &ChunkOptions::default());
let embedder = DeterministicEmbedder::new(1024);
let vectors = embedder.embed_chunks(&chunks).await?;
assert_eq!(vectors.len(), chunks.len());
# Ok(()) }

Capabilities

Embedder

infrastructure-embeddings — provider-agnostic text vectorization for semantic

NormalizePolicy

infrastructure-embeddings — provider-agnostic text vectorization for semantic

How to use it

From this crate's own rustdoc:

use infrastructure_embeddings::{chunk_text, ChunkOptions, DeterministicEmbedder, Embedder};
# async fn demo() -> Result<(), infrastructure_embeddings::EmbeddingError> {
let chunks = chunk_text("some long document text...", &ChunkOptions::default());
let embedder = DeterministicEmbedder::new(1024);
let vectors = embedder.embed_chunks(&chunks).await?;
assert_eq!(vectors.len(), chunks.len());
# Ok(()) }

Module structure

No public modules: the crate root is its whole surface.

Public surface

`crate root`

Re-exports. Exported here, defined elsewhere.

Boundary

Depends on no other workspace tier.

Shares tier infrastructure with 82 other crates: infrastructure-acquire, infrastructure-adapters-google-calendar, infrastructure-adapters-google-gmail, infrastructure-adapters-google-places, infrastructure-adapters-google-trends, infrastructure-adapters-shodan, infrastructure-adapters-yelp, infrastructure-agent, … (82 total).

_What this crate deliberately does NOT own is a judgment. No committed registry records one for it, so none is stated here._

Where it sits

Dependencies

Runtime, from outside the workspace.

Development, from outside the workspace.

Build. None.

Depended on by. 8 workspace crates.

Signal flow — what reaches this crate, and what it reaches.

flowchart LR
  n_application_legal_knowledge["application-legal-knowledge"] -->|uses| SELF
  n_application_wiki["application-wiki"] -->|uses| SELF
  n_domain_prose_shard["domain-prose-shard"] -->|uses| SELF
  n_foundation_text_clustering["foundation-text-clustering"] -->|uses| SELF
  n_infrastructure_doc_ingest["infrastructure-doc-ingest"] -->|uses| SELF
  n_infrastructure_html_text["infrastructure-html-text"] -->|uses| SELF
  n_infrastructure_web_ingest["infrastructure-web-ingest"] -->|uses| SELF
  n_tools_corpus["tools-corpus"] -->|uses| SELF
  SELF["infrastructure-embeddings"]
  classDef self fill:#1f883d,stroke:#1f883d,color:#fff;
  class SELF self;

Feature flags

flowchart LR
  n_default["default"]
  n_ort["ort"] --> n_dep_ort["dep:ort"]
  n_ort["ort"] --> n_dep_tokenizers["dep:tokenizers"]
  n_ort["ort"] --> n_dep_ndarray["dep:ndarray"]

Targets

Error model

No public error type was detected: no public item declares a type named *Error, and no public signature returns one.

Operational characteristics

No unsafe block, unsafe fn, unsafe impl or unsafe trait was found by the parser anywhere in this crate's source.

Configuration

Related capabilities

8 workspace crates depend on this one: application-legal-knowledge, application-wiki, domain-prose-shard, foundation-text-clustering, infrastructure-doc-ingest, infrastructure-html-text, infrastructure-web-ingest, tools-corpus.

Verification

Evidence by module. How often each public module is named by something executable.

What the tests establish, by name:

• empty_and_whitespace_yield_no_chunks — src/chunk.rs
• never_splits_a_multibyte_codepoint — src/chunk.rs
• overlap_ge_max_still_makes_progress — src/chunk.rs
• short_input_is_a_single_full_range_chunk — src/chunk.rs
• windows_overlap_and_cover_everything — src/chunk.rs
• batch_preserves_order_and_dim — src/dev.rs
• deterministic_and_unit_norm — src/dev.rs
• different_text_differs — src/dev.rs
• empty_is_finite_not_nan — src/dev.rs
• bge_m3_is_unit_dim_and_cross_lingual — src/ort.rs
• cuda_strict_probe — src/ort.rs

Documentation coverage

pie showData
    title Public items with rustdoc
    "Documented" : 2
    "No rustdoc detected" : 0

Metrics

pie showData
    title Public API by kind
    "enum" : 1
    "trait" : 1

pie showData
    title Rust source composition
    "Code" : 537
    "Blank or comment" : 248

Generation

Rendered by tools-corpus corpus readme from repository evidence alone, renderer schema 2, lexicon current. No model, network service or database was consulted. Regenerate with tools-corpus corpus readme --write; verify with --check.