The Five Enrichment Scopes: Taxonomy, Row, Column, Group, Cell

The most common question we get from a technical buyer is: "If I just want to refresh one field on one row, do I pay for the whole pipeline to run?" The answer is no, and the reason is the runner architecture. PYRAMYD's enrichment pipeline serves five distinct scopes from the same code path · taxonomy, row, column, group, and cell. They share the verification gate, the citation set, the embedding step, and the provenance write. They differ only on the atomic-unit boundary and the lock granularity.

Same pipeline · five granularities. A full-taxonomy refresh (2,606 categories) and a single-cell fix run through the same code. The difference is the advisory-lock scope and the work unit.

This is the architecture post · why the runner was designed for five scopes instead of one, how the advisory locks gate atomic runs, why the marginal-cost economics work, and what each scope is actually optimized for.

Section 1: Why five scopes

Enrichment platforms historically pick one scope and optimize for it. Bulk-only platforms (Fivetran, Airbyte) move whole tables. Per-row platforms (Hightouch, Census) move one record at a time. Per-field scrapers (Clay, Apollo) hit one column across many rows. None of those tools serves the operator who needs "refresh the demand field for these 12 categories before Tuesday's board meeting."

The five scopes map to five distinct operator workflows:

• Taxonomy · the quarterly hardening run. Refresh every row of every field group. Atomic, advisory-locked, runs once per quarter (or on schema migration). Cost: $3K-$6K per taxonomy depending on model tier.
• Row · the on-demand refresh. One node (e.g. one product, one category), all 10 field groups. The shape an analyst hits when they need a category 360 to be current before a customer meeting. Cost: $0.70-$1.40.
• Column · the new-dimension ship. One field across all rows in a taxonomy · e.g. ship a new compliance-status field to every product. Atomic per row, parallel across rows. Cost: scales with row count × per-cell cost.
• Group · the targeted refresh. One field group (e.g. all of "demand" or all of "compliance") across a row selection. Mid-grain · used when a Q1 demand metric needs to be re-pulled across the Top 50 categories.
• Cell · the surgical fix. One field on one row. Used when an operator spots a bad value or wants to refresh a single metric. Cost: $0.02-$0.08.

Section 2: The pipeline · shared across all five scopes

Every scope runs through the same five-stage pipeline. What changes is the work unit and the lock boundary; what doesn't change is the verification gate, the embedding step, the provenance write, and the audit log.

Stage 1: Source resolution

For each cell that will be enriched, the runner resolves a source set · web pages, API endpoints, internal docs, prior enrichment cache, peer-row references. The set is content-hashed per Stripe's idempotency pattern,[6] and if a recent enrichment exists with the same hash, the runner short-circuits and reuses it. This is the mechanism that lets us subsidize · the second customer asking the same question pays only the verification cost, not the full retrieval.

Stage 2: LLM enrichment

The selected frontier model (Opus 4.7 in the premium lane,[3] GPT-5 in the budget lane[4]) reads the source set and produces the cell value · plus a structured confidence score and citation set. The model is provider-agnostic per the twelve-factor backing-services pattern.[10]

Stage 3: Verification gate

A second model (typically cheaper, e.g. Haiku 4.5 or GPT-5 mini) verifies that the produced cell value is supported by the citations. This is where bad enrichments are caught before they write. The gate has a measured precision SLA · we publish quarterly. Cell values that fail the gate are marked needs-review and routed to a human queue, not silently dropped.

Stage 4: Embedding

Cells that pass the verification gate are embedded via Gemini Embedding-2[2] (1536-dim) and written to the pgvector store. This is what makes the enriched cell retrievable by APEX· and by every Studio surface that runs vector search.

Stage 5: Provenance write

The cell value is written to the canonical Postgres table along with a JSONB provenance blob · source URLs, retrieval timestamp, model name + version, prompt hash, verification score, embedding ID. This is the audit log the EU AI Act's transparency obligations require, and it's what makes the substrate trustable.

Section 3: Advisory locks and atomicity

The taxonomy scope · the "refresh everything" case · has to be atomic. If the runner crashes halfway through a 4,329-persona refresh, the graph can't be left in a state where 2,100 personas are fresh and 2,229 are stale. The lock is what guarantees this.

The runner uses Postgres advisory locks[1] · application-level mutex primitives that don't require a table or row to attach to. The taxonomy-scope lock is keyed by hashtext('enrichment_taxonomy_lock_' || taxonomy_name) and held for the duration of the run. If a second taxonomy-scope run is requested while the lock is held, it gets rejected immediately with a structured error · no queue, no race.

Row, column, group, and cell scopes don't take the taxonomy lock · they can run concurrently with each other and with a taxonomy run. The atomicity guarantee is therefore at the right granularity for each scope · taxonomy runs are serialized, smaller runs interleave.

Section 4: Why the economics work (marginal-cost subsidy)

Frontier-model inference cost dropped roughly 90% between Q4 2023 and Q4 2025 per a16z's AI-pricing analysis.[8] That curve made the "refresh whole taxonomies on a weekly cadence" workflow tractable. But the bigger economic lever is the cache-hit subsidy.

Every enrichment a customer runs writes to the shared cache. When the next customer asks the same question (which they do, often · enterprise software questions are highly repetitive across customers), the runner short-circuits at Stage 1 and reuses the verified result. Customer N+1 pays the verification cost only · roughly 15% of the original enrichment cost.

Across a typical tenant's usage pattern, cache hits land at 60-80% of cell-scope runs. That's the structural reason we price below retail LLM cost · the cost we incur is averaged across the whole customer base, not borne fully by each customer who asks.

Section 5: Runner telemetry · what production looks like

For concreteness, here's a snapshot from a recent taxonomy-scope run · 4,329 personas enriched across 10 field groups, frontier-tier model:[9]

• Concurrency cap: c=3 (rate-limit constrained by frontier model tier 1)
• Throughput: ~22 prompts/hour at steady state
• Wall-clock: 42 hours start-to-finish (split across 2 chained 24-hour Container Apps Jobs)
• Cache hit rate: 68% on the second run (the substrate stabilized between runs)
• Verification gate failure rate: 1.4% (mostly noisy source domains, routed to human queue)
• Total cost: ~$4,200 first run, ~$1,700 second run (cache subsidy)

Section 6: When to use each scope

Operator guidance · the heuristic we recommend:

• Cell when you spotted a bad value or need one number to be current before a meeting. 15-second turnaround.
• Row when you're about to use a category/product/persona in a customer deliverable and want every field group fresh. 90-second turnaround.
• Group when a quarterly metric (demand, compliance, economics) needs a targeted refresh across a row selection. 5-15 minute turnaround.
• Column when you're shipping a new dimension to a taxonomy · e.g. adding a new compliance status field. Hours to a day depending on row count.
• Taxonomy on a quarterly cadence or after schema migration. Atomic, lock-gated, 24-48 hour wall-clock.

The reason this matters for buyers: an enrichment platform that only serves one scope is always overpaying for the others. Bulk-only tools force you to refresh a whole taxonomy when you wanted one cell. Per-row tools force you to spin up a job per row when you wanted to batch. The five-scope runner is the "one tool, five workflows" pattern that eliminates that overhead · which is ultimately why the marginal-cost economics work for everyone on the platform.