How synthetic commerce data is generated¶

This guide explains the process implemented in recommendations.synthetic, from a configuration to a verified relational source that the recommendation worker can read. It follows generator v5, profile version v1, manifest version v3, and schema generated-source-v3.

The generator is a deterministic program: it constructs Catalog metadata, schedules shopping contexts, chooses Items, and emits rows. It does not use an LLM, train a generative model, import merchant transaction records, or reconstruct real people. Its purpose is to create controlled inputs for testing recommendation behavior and exercising the training pipeline at different scales.

Each generated browsing session and purchase order receives one deterministic coarse country/region/city path from a fixed synthetic hierarchy. View, online-purchase, and offline-purchase rows carry that complete path; the canonical source queries expose it to the training adapter. The hierarchy is source-local test data, not a geocoder or a claim about a real Shopper's location. Geographic fields participate in the v3 logical table digests, so materialization verification covers them.

Catalog names, brands, and departments come from the EPA ENERGY STAR Model Index, downloaded on 2026-09-23. The bundled snapshot has 45,175 source rows; the development Catalog uses 25,000 unique product titles across 1,286 source brand names and 43 categories. Product titles are assembled from source brand, model name, and model number fields. The index contains currently certified models, which does not guarantee that a product is currently in stock at a retailer. The source fields are public domain under the EPA data license. The large central-air-conditioner category is excluded from the snapshot selection. Prices, eligibility, creation times, product IDs, and shopping activity remain generated by the harness.

Read this guide in order for the full explanation. For commands and Docker configuration, keep the generated-source operations guide alongside it. The domain vocabulary defines the capitalized terms used here.

For a shorter reading path:

Start with the overall flow and the five-table model.
Learn the mechanics through repeatable randomness, Catalog cohorts, group schedules, and Item selection.
Follow content identity into database loading.
Compare live simulation, then try the reproducibility experiment.
Use the evidence map and source-code map as references.

1. Start with the three different processes¶

There are three connected activities in this repository:

Activity	Input	Output	Main question
Generate a relational source	Profile, seed, scope, cutoff, capabilities	Five logical tables, manifests, verified database realization	Can we reproduce these exact source contents?
Run a commerce simulation	Seed Catalog, scenario, bounded visitor plans	New views and Orders in a mutable Simulation Data Source	What happens as controlled shopping activity accumulates?
Train and serve recommendations	Registered source and training configuration	Published Recommendation Snapshot and API responses	What recommendations does the service derive from that evidence?

Generating data does not submit a Training Run. Verifying a database does not execute the serving oracles. A simulation has its own state and lineage; its changing source contents cannot continue to claim the original generated dataset's identity.

flowchart TD
    Config["Profile + seed + scope + cutoff + capabilities"] --> Generator["Deterministic source generator"]
    Generator --> Rows["Five logical row streams"]
    Generator --> Manifest["Generation Manifest"]
    Generator --> Oracle["Oracle Manifest"]
    Rows --> Load["SQLite or PostgreSQL materializer"]
    Load --> Verify["Read back ordered rows and verify"]
    Manifest --> Verify
    Verify --> Source["Verified Generated Relational Data Source"]
    Verify --> Receipt["Materialization Receipt"]
    Source --> Adapter["Normal Data Source Adapter"]
    Adapter --> Training["Training Run"]
    Training --> Snapshot["Published Recommendation Snapshot"]
    Snapshot --> Serving["Serving API"]
    Serving --> Evaluate["Separate outcome evaluation"]
    Oracle --> Evaluate
    Generator --> Seed["Catalog and compatibility seed"]
    Seed --> Simulation["Mutable Simulation Data Source"]
    Visitors["Storefront actions and Synthetic Visitors"] --> Simulation
    Simulation --> Adapter

The generator owns source provisioning. The recommendation worker reads that source through the ordinary adapter boundary; serving reads published snapshots. These ownership boundaries also explain the privacy rule: synthetic interaction rows belong in the generated source, while manifests and evidence contain counts, digests, and bounded observations rather than row dumps.

2. Resolve the recipe before producing rows¶

The entry point is cli.main. It builds a GenerationConfig, which normalizes identifiers, resolves the named profile, normalizes the cutoff to UTC, and validates the seed and capability flags. A seed must be a non-negative integer; a Boolean is not accepted as an integer seed.

The defaults are:

Input	Default	What it controls
Data Source ID	`generated-source`	Identity of the registered source
Tracking ID	`synthetic-property`	Commerce Property boundary within the source
Catalog ID	`synthetic-catalog`	Catalog boundary within that property
Profile	`smoke`	Row quotas, history length, category count
Root seed	`20260805`	Seeds for the deterministic random streams
Cutoff	Midnight UTC on the current date	Exclusive end of the interaction history
Offline Purchases	Enabled	Whether purchases are split into two channels
Compatibility rules	Enabled	Whether explicit complement rules are generated
Maximum cart items	Unset	Optional cap on purchase rows per Order (`--max-cart-items`)
Maximum view pages per session	Unset	Optional cap on view rows per Browsing Session (`--max-view-pages-per-session`)

The logical window is half-open:

history_start = cutoff - profile.history_days
history_start <= event_at < cutoff

The omitted cutoff is resolved to midnight UTC on the current date when the configuration is created. The resolved value is recorded in the Generation Manifest. Pass an explicit --cutoff (or Python cutoff) to reproduce the same logical history on a later run.

The released profiles are:

Profile	Catalog Items	View rows	Purchase rows, all channels	History	Categories
`smoke`	1,000	25,000	10,000	90 days	20
`development`	25,000	100,000,000	20,000,000	365 days	43
`qualification`	200,000	100,000,000	5,000,000	730 days	43

These are physical row counts, not numbers of distinct sessions, Orders, purchased units, or unique interacting Items. For example, a three-Item Order normally produces at least three purchase rows; repeated lines add more rows without adding another distinct Item to the Order.

The CLI exposes smoke and development. The Python API also accepts qualification and custom GenerationProfile objects. A reduced profile is useful for experiments, but its results do not establish performance for the unmodified named profile. Custom sizes also need enough Catalog Items to satisfy the reserved cohorts and distinct-Item group sizes; positive sizes alone are not a guarantee that an arbitrary custom profile will be workable.

GenerationConfig.overrides and calibration are recorded lineage fields. The current generator does not interpret those mappings as instructions to change distributions. Actual row counts come from the resolved profile. Do not mistake a recorded calibration mapping for a fitted model.

Both group limits must be positive integers. Setting either limit changes the generation recipe, so the requested limit is recorded in the Generation Manifest and the result cannot make an unmodified profile qualification claim. The limits apply to physical rows, including repeated views or purchase lines. The total row quotas stay fixed. Append does not accept custom limits.

3. Understand the five-table data model¶

Every logical source exposes these tables, even when an optional table is empty:

Table	One row means	Significant fields
`catalog`	One Item's current metadata and Recommendation Eligibility	Scope identifiers, Product ID, category, brand, price, creation time, title, description
`views`	One Item view within a Browsing Session	Scope identifiers, Product ID, Session ID, event time, source sequence
`online_purchases`	One Item line in an Online Order	Scope identifiers, Product ID, Order ID, event time, quantity, source sequence
`offline_purchases`	One Item line in an Offline Order	Same logical columns as Online Purchases
`compatibility_rules`	A directed complement rule from one Item to another	Scope identifiers, anchor Product ID, candidate Product ID

erDiagram
    CATALOG ||--o{ VIEWS : "Item viewed"
    CATALOG ||--o{ ONLINE_PURCHASES : "Item purchased online"
    CATALOG ||--o{ OFFLINE_PURCHASES : "Item purchased offline"
    CATALOG ||--o{ COMPATIBILITY_RULES : "anchor reference"
    CATALOG ||--o{ COMPATIBILITY_RULES : "candidate reference"

    CATALOG {
        string tracking_id PK
        string catalog_id PK
        string product_id PK
        boolean is_eligible
        string category_id
        string brand
        decimal price
        datetime created_at
        string title
        string description
    }
    VIEWS {
        integer source_sequence PK
        string tracking_id FK
        string catalog_id FK
        string product_id FK
        string session_id
        datetime event_at
    }
    ONLINE_PURCHASES {
        integer source_sequence PK
        string tracking_id FK
        string catalog_id FK
        string product_id FK
        string order_id
        datetime event_at
        decimal quantity
    }
    OFFLINE_PURCHASES {
        integer source_sequence PK
        string tracking_id FK
        string catalog_id FK
        string product_id FK
        string order_id
        datetime event_at
        decimal quantity
    }
    COMPATIBILITY_RULES {
        string tracking_id PK,FK
        string catalog_id PK,FK
        string anchor_product_id PK,FK
        string product_id PK,FK
    }

The foreign keys are composite: (tracking_id, catalog_id, product_id) refers to the matching Catalog row. Compatibility has two such references. The ER diagram shows relationships, not all physical database types or check constraints.

Data Source ID is configuration/manifest identity; it is not repeated in each generated table. The source connection plus Tracking ID and Catalog ID form the complete Commerce Scope. Product IDs alone are not globally unique.

There are no separate session-header or Order-header tables in this model. Shared Session IDs or Order IDs group interaction rows. source_sequence is a zero-based counter local to each interaction table; it is not a Shopper ID or a timestamp.

4. Make randomness repeatable¶

The generator derives a separate seed for each named concern:

substream_seed = big_endian_integer(
    first_8_bytes(SHA3-512(UTF8(root_seed + ":" + stream_name)))
)

The implementation formats the integer seed as text in that expression. Its random algorithm label is MT19937/SHA3-512-substreams: SHA3-512 derives seeds, and Python's random.Random supplies the pseudorandom draws for interaction generation.

flowchart LR
    Root["Root seed"] --> Hash["SHA3-512 of seed and stream name"]
    Hash --> C["catalog seed recorded"]
    Hash --> V["views seed"]
    Hash --> O["online_purchases seed"]
    Hash --> F["offline_purchases seed"]
    Hash --> R["compatibility seed recorded"]
    V --> VR["Fresh RNG for each views iteration"]
    O --> OR["Fresh RNG for each online iteration"]
    F --> FR["Fresh RNG for each offline iteration"]
    VR --> VS["Reproducible view stream"]
    OR --> OS["Reproducible online stream"]
    FR --> FS["Reproducible offline stream"]

Catalog metadata and compatibility rules currently use arithmetic construction, so their recorded substream seeds do not imply random draws are consumed. Changing only the seed changes interaction choices and times, while the Catalog and compatibility tables remain unchanged.

Each call to iter_table starts the relevant interaction RNG again. It regenerates the same rows instead of caching an entire Interaction Dataset. Different tables do not advance one shared global RNG. Within a table, however, time selection, product selection, and duplicate retries share the stream: adding a random draw can change everything generated after it.

To reproduce a source, preserve the implementation version, resolved profile, seed, scope, time bounds, capability flags, and runtime/dependency environment. The seed alone is insufficient. Changing worker count or insert batch size should affect execution, not logical rows or digests.

5. Construct the Catalog and its cohorts¶

GeneratedRelationalDataSource.__init__ first calls _build_catalog. The Catalog is retained in memory, together with lookup maps and cohort sets; interaction history is streamed later.

The Catalog starts with 17 named signal Product IDs and fills the remaining quota with numbered Product IDs. All IDs are sorted before assigning metadata. This makes construction stable and gives subsequent algorithms a reproducible index for each Item.

The main cohorts are assigned using quotas and reserved signal Items:

Cohort or property	Released-profile share	Smoke count	Purpose
Eligible Items	95%	950	Items potentially usable in recommendations
Ineligible Items	5%	50	Exercise filtering despite source references
Strict Cold cohort	5%	50	Eligible Items with complete metadata and excluded from interaction selection
Sparse cohort	15%	150	Items excluded from ordinary warm selection
Complete descriptive metadata	80%	800	All five descriptive fields populated
Partial descriptive metadata	15%	150	Some descriptive fields populated
Missing descriptive metadata	5%	50	All five descriptive fields absent

Eligibility and metadata completeness are different dimensions. Do not add all the percentages in this table together. The descriptive fields are category, brand, price, title, and description; creation time is assigned separately, including on Items with no descriptive metadata.

Strict Cold, sparse, and ineligible cohort assignment is disjoint in the current construction. Strict Cold candidates are deliberately chosen from complete-metadata positions. The remaining eligible Items form _warm_ids, the population available to ordinary interaction selection.

There is an implementation limit behind the word "sparse": only the reserved sparse signal Item is explicitly inserted into group index 10 in each nonempty interaction stream. The other Items in the designated sparse cohort are excluded from ordinary sampling and receive no interactions. The cohort label therefore does not mean every one of those Items satisfies the domain definition of a Sparse Item, which requires some observed evidence.

For zero-based sorted Catalog index i, the main metadata formulas are:

category_id     = normalized EPA product category
brand           = EPA brand name
title           = brand + model name + model number when needed to disambiguate
description     = ENERGY STAR category and model identifier
price           = (10 + ((i * 17) modulo 10000)) / 100
created_at      = history_start - (30 + (i modulo 90)) days

Prices use Decimal, rounded to four decimal places. The model title and department identify a real manufacturer model; the description is a concise label derived from the EPA model-index fields, not manufacturer marketing copy. Positions 0–15 in each block of 20 have complete metadata; positions 16–18 have partial metadata; position 19 has missing descriptive metadata.

Some Strict Cold Items get a creation time 15–34 days before cutoff, while others are older than the history window. This illustrates that "cold" describes absent interactions, not necessarily a recent creation date.

6. Plan group sizes before choosing Items¶

Co-occurrence needs contexts: Also Viewed groups views by Browsing Session, and Frequently Bought Together groups purchases by Order. _group_schedule first divides an exact row budget into distinct Item appearances within groups and repeated appearances within those same groups.

For a stream with N rows:

repeated_rows = floor(N * 5 / 100)
distinct_appearances = N - repeated_rows

"Distinct appearances" is the sum of distinct Items per group, not the number of distinct Items across the whole table. An Item appearing in ten different sessions contributes ten appearances.

flowchart TD
    N["Exact physical row quota"] --> Split["Reserve 5 percent repeated rows"]
    Split --> Distinct["Budget for distinct Items within groups"]
    Split --> Repeats["Repeated-row budget"]
    Distinct --> Oversized["Reserve deliberate oversized groups"]
    Oversized --> Regular["Fit repeating group-size cycle"]
    Regular --> Last["Adjust final regular group to close quota"]
    Last --> Groups["Compact group schedule"]
    Repeats --> Spread["Distribute repeats across all groups"]
    Spread --> Groups
    Groups --> Choose["Choose distinct Items for each group"]
    Choose --> Emit["Emit distinct rows, then repeated rows"]

Regular group sizes follow these cycles:

Views:     1, 2, 3, 4, 1, 2, 3, 5, 1, 2, 3, 6, 4, 5, 6, 7, 8, 9, 20, 30
Purchases: 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 4, 5, 7, 10, 20

The first three regular groups are forced to size three for planted relationships. The final regular group absorbs the arithmetic remainder, staying within 100 distinct Items for views or 200 for purchases on the released profiles. Two additional groups deliberately exceed those boundaries: 101 and 200 distinct Items for views, and 201 and 400 for purchases. These give the training pipeline oversized contexts on which to exercise its group limits.

For G total groups, group g receives:

floor(repeated_rows / G) + (1 if g < repeated_rows modulo G else 0)

The generator repeats Items from the start of that group's selected list. Repeats have later timestamps and new source sequences; they are repeated Item evidence, not identical row copies. They exercise the distinction between physical event counts and distinct Items in a context.

With a group limit, the generator divides rows evenly among enough sessions or Orders to respect the cap and omits the deliberately oversized groups. It retains the 5% repeated-row quota where the cap is at least two. A cap of one creates single-row contexts with no repeated rows.

The schedule itself is compact. _base_group_size_sum sums full cycles and a short remainder arithmetically instead of visiting millions of groups. The search checks a bounded neighborhood around the cycle-derived estimate. Generator v3 uses the actual purchase-cycle mean of 3.55; an older estimate of 3.7 could leave an impossible final group at large scales. The schedule tests protect exact quotas and bounded groups.

Worked smoke row budget¶

With Offline Purchases enabled, 80% of purchase rows are online and the remainder are offline:

Stream	Physical rows	Distinct appearances within groups	Repeated rows
Views	25,000	23,750	1,250
Online Purchases	8,000	7,600	400
Offline Purchases	2,000	1,900	100

Disabling Offline Purchases moves the entire purchase quota to the online stream; it does not discard 20% of the requested purchases. The offline table remains present and empty, and its optional adapter query is omitted.

7. Choose Items and plant recognizable behavior¶

For each group, _products_for_group chooses a list of distinct Product IDs. The initial groups contain deliberately paired signal Items:

Stream	First three groups each contain
Views	`view_anchor`, `view_candidate`, `metadata_twin` signal Items
Each purchase channel	`fbt_anchor`, `fbt_candidate`, `category_complement` signal Items

These names abbreviate the product-signal-* identifiers in the source. Repeating a pair across separate contexts establishes co-occurrence support. Repeating the same Item within just one context does not create another independent session or Order.

At group index 10, the reserved sparse signal is inserted after filling the other distinct positions with ordinary selections. Other groups use _choose_product, whose decisions run in the following priority order:

flowchart TD
    Start["Choose next distinct candidate"] --> Ineligible{"Planted selection enabled and ordinal mod 100 = 7?"}
    Ineligible -->|Yes, creation time permits| I["Reserved ineligible Item"]
    Ineligible -->|No| Viewed{"View and ordinal mod 97 = 11?"}
    Viewed -->|Yes| V["Most-viewed signal"]
    Viewed -->|No| Bought{"Purchase and ordinal mod 83 = 13?"}
    Bought -->|Yes| B["Best-seller signal"]
    Bought -->|No| Rising{"View in last 7 days and ordinal mod 113 = 3?"}
    Rising -->|Yes| R["Rising signal"]
    Rising -->|No| Warm["Sample warm population with head bias"]
    Warm --> Time{"Item created by group time?"}
    Time -->|Yes| Candidate["Candidate"]
    Time -->|No| Retry["Retry up to 25 draws, then baseline Item"]
    I --> Candidate
    V --> Candidate
    B --> Candidate
    R --> Candidate
    Retry --> Candidate
    Candidate --> Unique{"Already in this group?"}
    Unique -->|No| Append["Append Item"]
    Unique -->|Yes| Again["Try a selection with planted branches disabled"]
    Again --> Unique

The diagram condenses the loop: after an unsuccessful duplicate retry, the implementation returns to its normal selection loop. The 25-attempt bound applies to creation-time sampling inside one _choose_product call, not to the entire group uniqueness loop. Named profiles provide ample candidate populations; shrinking a custom Catalog below a requested distinct group size is unsafe.

Ordinary selection uses a mixture, where S is the warm population size and U is a uniform draw:

15% of choices: uniform integer from 0 through S - 1
85% of choices: min(S - 1, floor(U ** 2.35 * S))

Raising a value between zero and one to 2.35 moves it toward zero, concentrating draws on earlier Catalog positions. The uniform component still gives the tail opportunities. Although the helper is named _zipf_index, this is a bounded head-biased mixture, not an exact Zipf distribution fitted to merchant traffic. Uniqueness retries and explicit signal insertions further affect the final distribution.

The periodic planted branches are not exact final percentage quotas. They depend on the current row ordinal, group sizes, duplicate rejection, and branch priority. The reserved signal Items are also members of the appropriate ordinary populations unless excluded by their cohort.

Signal names are not behavioral guarantees¶

The Generation Manifest lists all 17 signal roles, but not every role has a dedicated behavioral construction. In particular, metadata_twin does not receive copied metadata, category_complement does not receive a special category assignment, and steady, declining, tie_a, and tie_b do not get explicitly balanced or time-shaped interaction schedules. They receive the general metadata/selection treatment where applicable.

Treat these as lineage labels until a specific source invariant and serving observation establish the intended effect. An Item named best_seller is not, by its name alone, proof of a top ranking.

8. Add logical time, quantities, and channel identity¶

Time within the history window¶

_group_time computes a group's base time from its index g and two random draws:

day = (g * multiplier + random integer in [0, 30]) modulo history_days
second = (g * 7919 + random integer in [0, 86399]) modulo 86400
multiplier = 37 for views, 53 for purchases
event_base = history_start + day days + second seconds

On the final history day, the base second is capped at 86400 - 500, leaving room for the released profiles' largest groups. Distinct Items are emitted one second apart; repeated rows follow them at subsequent seconds. Normal sampling checks creation time before selecting an Item. Strict Cold Items are excluded from interaction sampling altogether.

The resulting table is grouped by session/Order identity, not globally chronological. A later session can occur on an earlier day. This is intentional: canonical ordering lets the source adapter reduce one complete shopping context at a time.

Purchase quantities¶

Quantities use a global purchase-row ordinal across online first, then offline:

Position in the combined purchase quota	Quantity	Smoke rows
First 85%	`1.000000`	8,500
Next 10%	`2.000000`	1,000
Next 4%	Integer 3, 4, or 5, selected by ordinal modulo 3	400
Final 1%	Cycle of 1.25, 1.5, 1.75, 2.25, 2.5	100

These are deterministic bands, not shuffled probability draws. With the default 80/20 channel split, all online rows have quantity one; the later quantity bands appear in the offline stream. That is an actual property of this generator, not a claim about real channel behavior. All quantities use Decimal with six decimal places, preserving fractional purchases without binary floating-point rounding.

Deliberate Order ID collisions¶

Normally, offline Order IDs use a large numeric offset from online Order IDs. The first max(1, floor(offline_group_count / 100)) offline groups deliberately reuse the corresponding online source Order IDs. Correct consumption still keeps those Orders separate because purchase identity includes Purchase Channel in addition to Commerce Scope.

The generated PurchaseRow carries the channel in memory. In the physical schema, the two table names identify the channels; the adapter reconstructs channel identity when reading them.

9. Generate explicit compatibility rules¶

_iter_compatibility takes the first 20% of sorted eligible Items as anchors and creates three distinct directed candidates per anchor. Candidate offsets advance by multiples of seven through the eligible list, wrapping around when necessary. Self-links and duplicate candidates are avoided.

The first anchor includes the reserved compatibility signal. A deliberately small subset of rules instead targets the reserved ineligible Item: floor(total_rule_count * 2 / 100) rules. This tests that an explicit relationship cannot override Recommendation Eligibility.

For smoke, this yields 950 * 20% = 190 anchors and 190 * 3 = 570 rules, of which 11 target the ineligible signal. Floor arithmetic means the resulting share is close to, rather than exactly, 2%. Rules are sorted by scope, anchor, and candidate before emission. They are directed; the generator does not automatically add a reverse rule.

Disabling compatibility leaves an empty table and omits the optional adapter query. Compatibility generation is deterministic arithmetic, independent of the interaction RNG streams.

10. Give the contents a verifiable identity¶

Two databases can contain the same logical rows while having different files, page layouts, index layouts, or insertion order. The generator therefore identifies logical content through a canonical serialization, implemented in manifest.py.

Canonical order and row bytes¶

Table	Canonical ordering after the scope identifiers
Catalog	Product ID
Views	Session ID, event time, Product ID, source sequence
Each purchase channel	Order ID, event time, Product ID, source sequence
Compatibility	Anchor Product ID, candidate Product ID

Each row is converted to a fixed-column JSON array. Strings use Unicode NFC normalization; timestamps use UTC with six fractional digits and Z; Decimals become non-exponent strings; Booleans and nulls retain their JSON types. The array is encoded as compact UTF-8 JSON followed by one LF newline. Decimal scale is retained for nonzero values, so 1.2500 is not rewritten to 1.25. Generation and database readback must agree on these bytes.

flowchart LR
    Row["Ordered logical row"] --> Values["Fixed column order"]
    Values --> Normalize["Canonical strings, timestamps and decimals"]
    Normalize --> Bytes["UTF-8 JSON array + newline"]
    Bytes --> TableHash["Incremental SHA3-512 per table"]
    TableHash --> Identity["Schema + fixed table order + counts + digests"]
    Identity --> Dataset["SHA3-512 dataset_id"]

Hashing is incremental: it updates one digest as rows pass through, without collecting all their serialized bytes. Empty tables have zero rows and the digest of an empty byte stream.

dataset_id hashes the schema version and all five table names, counts, and digests in fixed order. It does not hash the entire Generation Manifest. Metadata such as generator version and root seed explains lineage, while the table digests identify the resulting content. In particular, changing only Data Source ID changes manifest identity fields but not table contents or dataset_id; Tracking ID and Catalog ID are in the rows and therefore affect the content hash.

Three artifacts, three different questions¶

Artifact	What it records	What it establishes
`generation-manifest.json`	Versions, requested/resolved configuration, seeds, scope, capabilities, table counts/digests, signal roles, oracle checksum	The logical source identity and its generation recipe
`oracle-manifest.json`	Versioned expected observations	What a separate evaluator should check
`materialization-receipt.json`	Database engine, materialization ID/time, observed counts/digests, constraint checks, verification status	Whether this database realization matches the Generation Manifest

The current Oracle Manifest contains exactly five observations: Catalog row count, view row count, total purchase row count, inclusion of the FBT candidate for its anchor, and inclusion of the Also Viewed candidate for its anchor. It does not contain a complete suite of ranking, fallback, cold-start, or temporal assertions.

The materializers do not run training or evaluate the two source-to-serving observations. A receipt with verified: true proves materialization integrity, not recommendation correctness. Database readback is independent of the producer's digest, but both still describe what this generator produced; a consistent generator defect can pass a content-integrity check.

Receipts intentionally vary across materializations because their UUID and wall-clock creation time vary. File-backed SQLite can also record a physical file digest. PostgreSQL records no physical database digest. Compare logical table digests and dataset_id for reproducibility, rather than expecting receipts or database files to be byte-identical.

11. Materialize the streams in a database¶

Both materializers create the five-table contract with primary keys, foreign keys, validity checks, and canonical-order indexes. Both read back rows in canonical order, recompute counts and digests, and return a MaterializedSource containing engine, queries, and receipt on successful verification.

SQLite: a straightforward local path¶

The CLI first calls generate_relational_data_source. Construction builds the Catalog and walks every logical table to produce the Generation Manifest. It then calls materialize_sqlite, which regenerates the interaction streams for insertion in bounded batches, followed by a database readback. Thus the local path has two generation traversals plus the verification read.

SQLite stores decimals and timestamps as canonical text and eligibility as an integer Boolean. Readback restores the Boolean interpretation for hashing. By default the database is generated-source.sqlite in the output directory; Python callers can also use an in-memory source.

Existing generated tables are rejected unless replace=True / --replace is explicit. Unlike the PostgreSQL path below, SQLite operates directly on canonical table names and has no verified staging-table swap. Its receipt comparison occurs after the load transaction exits. Do not infer PostgreSQL's failed-replacement preservation guarantee from the SQLite implementation.

PostgreSQL: generate once, load concurrently, publish after verification¶

The PostgreSQL CLI calls provision_postgresql. It defers the Generation Manifest until rows have been streamed for COPY, allowing the producer to compute table counts and hashes during the same pass that supplies the database.

sequenceDiagram
    participant CLI as Generator CLI
    participant G as Logical generator
    participant W as COPY workers
    participant DB as PostgreSQL
    participant V as Verification workers
    CLI->>G: Resolve config, build Catalog, defer manifest
    CLI->>DB: Check destination, create unique staging tables
    loop Each table, Catalog first
        G->>G: Generate ordered rows, count and hash
        G->>W: Enqueue bounded batches
        W->>DB: COPY batches into staging table
        W-->>CLI: All writers complete for this table
    end
    CLI->>G: Complete Generation Manifest from captured hashes
    CLI->>DB: Build canonical-order indexes in parallel
    CLI->>V: Verify tables in parallel
    V->>DB: Read rows in canonical order
    V-->>CLI: Recomputed counts and digests
    CLI->>DB: Inspect constraints and compare evidence
    alt Everything matches
        CLI->>DB: Atomically publish all five canonical table names
        CLI->>CLI: Write manifests and verified receipt
    else Load or verification fails
        CLI->>DB: Best-effort staging cleanup
        CLI->>CLI: Fail and retain previously published tables
    end

Tables are produced sequentially, beginning with Catalog so dependent rows have valid foreign keys. For each table, one Python producer feeds a shared bounded queue and several COPY workers. Any available worker can take the next batch. Physical insertion order can differ from generation order; verification restores canonical order through SQL ORDER BY.

The default worker count is four, with a supported range of 1–16. Effective COPY concurrency also considers the number of batches and a caller-supplied connection pool's persistent capacity. The queue holds at most one pending batch per active worker. There can additionally be one batch being processed per worker and a batch being assembled by the producer. With W workers and batch size B, those Python row buffers are roughly bounded by (2W + 1) * B, excluding driver buffers, the Catalog, current group, and database-side memory.

This is parallel database writing, not parallel product selection. Increasing --workers cannot remove the cost of the single producer, per-row hashing, index builds, or database readback. Index construction and verification parallelize across tables separately. One-pass provisioning still reads the entire database content afterward to verify it.

The staging names include a random suffix used only for operational isolation. Publication is one transaction: recheck destination policy, drop old generated tables if replacement was requested, and rename all verified staging tables and indexes. A load, index, or receipt failure before publication leaves the previous canonical source intact. Cleanup is best effort; an unsuccessful cleanup may leave uniquely named, unpublished staging tables.

The CLI writes evidence files after successful materialization. Database publication and the three filesystem writes are not one distributed transaction: an artifact-write failure can leave a successfully published database with missing or partial evidence files. Do not assume the database is absent simply because the command failed while writing its artifacts.

Direct Python callers can also pass an already manifested source to materialize_postgresql; that retains the earlier manifest-generation pass. Use provision_postgresql for the integrated single-generation-pass path.

12. Follow generated evidence into recommendation training¶

Materialization does not need a special ingestion endpoint. source.canonical_queries() supplies the same columns, scope filters, time filters, and ordering expected by SqlAlchemySourceAdapter. To use the source in a running service, register its connection and queries, allow its exact Tracking ID/Catalog ID scope, and submit a normal Training Run. See the runtime setup and technical implementation for that configuration.

flowchart LR
    DB["Generated relational tables"] --> Read["Scoped consistent source read"]
    Read --> Groups["Stream and reduce sessions and Orders"]
    Groups --> Counts["Bounded aggregate co-occurrence and popularity evidence"]
    Read --> Metadata["Catalog metadata and compatibility"]
    Counts --> Build["Build recommendation strategies"]
    Metadata --> Build
    Build --> Publish["Atomically publish complete snapshot"]
    Publish --> Serve["Serve from snapshot"]
    Serve --> Observe["Compare prescribed observable outcomes"]

The generated source remains the source of raw rows. The worker's derived aggregate staging does not become an archive of those rows. A verified Generated Relational Data Source is useful because input identity stays fixed while training configuration or service code changes.

Generation-time history bounds and Training Run evidence bounds are separate inputs. In particular, a service training against a later current-time window may include only part of the generated history or none of it. Align the experiment's logical time, source history, training window, and prescribed observations before interpreting empty results as a scoring defect. The generator CLI does not set the worker's cutoff.

13. Understand how live simulation differs¶

The Commerce Simulation Harness reuses generated Catalog and compatibility content through SimulationRunSeed.from_generated_source in simulation/source.py. It keeps their seed lineage, but does not copy the generated historical views and purchases into a Simulation Run.

Instead, scenario.py resolves a scenario and precomputes visitor plans. For each visitor it:

Selects a Synthetic Shopping Mission using the configured weights.
Forms a sorted, deduplicated candidate pool from the mission's Catalog cohorts.
Chooses a view count within the mission's limits and samples that many distinct Items.
Draws whether checkout occurs using the mission's checkout probability.
Plans one view action per selected Item and, if selected, checkout of those Items.
Assigns deterministic logical timestamps, session/Order identifiers, and think times.

Simulation seeds derive from the resolved scenario digest, concern name, and visitor/action coordinates using SHA3-512. This is a different seed scheme from the bulk source generator's SHA3-512 table substreams. Separate coordinates make planned visitor decisions independent of the order in which runtime threads happen to execute them.

flowchart TD
    Scenario["Authored scenario + defaults + allowed overrides + hard caps"] --> Resolve["Resolved scenario identity"]
    Resolve --> Plan["Mission, Items, checkout, IDs and logical action times"]
    Plan --> Phases["Partition actions at training checkpoints"]
    Phases --> Runner["Bounded Synthetic Visitor Runner"]
    Runner --> HTTP["Same storefront HTTP actions as human browsing"]
    HTTP --> Mutable["Append views and Online Purchases"]
    Mutable --> Stop["Checkpoint: close admission and drain writes"]
    Stop --> Train["Submit normal Training Run and await snapshot"]
    Train --> More{"Intermediate checkpoint?"}
    More -->|Yes| Runner
    More -->|No| Report["Final scenario evidence"]

The default independent scenario defines 20 visitors, mission weights of 3:1, and checkpoints after visitors 10 and 20. Its first mission chooses 2–3 views from two categories and a 0.75 checkout probability; its second chooses 1–2 views from another category and a 0.25 checkout probability. Those weights and probabilities describe draws, not exact quotas of 15 visitors or a guaranteed conversion rate.

An Independent-Behavior Scenario keeps selections independent of served recommendations. A Recommendation-Feedback Scenario may follow served Items, making subsequent evidence partly a consequence of earlier recommendations. That feedback can be studied, but cannot be treated as independent confirmation of recommendation quality. Neither scenario measures real business uplift.

The optional explicit personalization journey machinery is another layer, covered in the personalization guide; it is not part of the bulk five-table source generator described above.

14. Try a small reproducibility experiment¶

Use make setup if the locked development environment is not already installed. Start with smoke: development contains 120 million interaction rows and requires substantially more time and storage. Choose new output directories for each run; existing source tables are rejected without --replace.

uv run --frozen recommendations-generate-source \
  --profile smoke --seed 47 --output-dir artifacts/generation-lesson-a

uv run --frozen recommendations-generate-source \
  --profile smoke --seed 47 --output-dir artifacts/generation-lesson-b

cmp artifacts/generation-lesson-a/generation-manifest.json \
    artifacts/generation-lesson-b/generation-manifest.json

cmp artifacts/generation-lesson-a/oracle-manifest.json \
    artifacts/generation-lesson-b/oracle-manifest.json

Successful cmp commands produce no output. Both runs should report the same dataset_id. Inspect the JSON files to find versions, resolved quotas, table digests, and verification status. Expect the receipts' operational identities and timestamps to differ.

If the sqlite3 command is available, inspect aggregate structure without dumping interactions:

sqlite3 artifacts/generation-lesson-a/generated-source.sqlite '
SELECT is_eligible, COUNT(*) FROM catalog GROUP BY is_eligible;
SELECT COUNT(*) AS view_rows FROM views;
SELECT COUNT(*) AS online_rows FROM online_purchases;
SELECT COUNT(*) AS offline_rows FROM offline_purchases;
SELECT COUNT(*) AS compatibility_rows FROM compatibility_rules;
SELECT quantity, COUNT(*) FROM (
  SELECT quantity FROM online_purchases
  UNION ALL
  SELECT quantity FROM offline_purchases
) GROUP BY quantity ORDER BY CAST(quantity AS NUMERIC);
'

The expected counts are 950 eligible and 50 ineligible Items, 25,000 views, 8,000 online rows, 2,000 offline rows, and 570 rules. Quantity bands should total 8,500 rows at one, 1,000 at two, 400 across integers three through five, and 100 fractional rows.

Useful follow-up experiments, each in a fresh directory:

Change	What to compare	Expected lesson
Use seed 48	Per-table logical digests	Interaction tables change; arithmetic Catalog and rules stay the same
Add `--no-offline-purchases`	Channel counts and online digest	Offline becomes empty; online now contains all 10,000 purchase rows
Add `--no-compatibility`	Rule count and dataset ID	The fifth table remains part of identity even when empty
Change `--cutoff`	Time bounds and table digests	A reproducible experiment includes time, not just randomness
Load the same config into PostgreSQL	Manifest and receipt logical digests	Physical engine choice should not alter logical content
Change PostgreSQL `--workers` or `--batch-size`	Manifest versus operational duration	Execution settings should preserve content identity

Use the PostgreSQL setup instructions for the last two experiments. Replacement must target an explicitly replaceable destination. The complete CLI option reference is available with recommendations-generate-source --help.

15. Know what the tests and evidence establish¶

The most relevant executable evidence is:

Tests	Contract they exercise
`test_synthetic_source.py`	Repeated generation equality, counts, ordering, time bounds, metadata/eligibility quotas, quantities, manifests, verified SQLite and adapter reads
`test_synthetic_schedule.py`	Schedule compatibility, bounded scheduling work, all named purchase-profile quotas with/without offline, completion with large purchase quotas
`test_synthetic_serialization.py`	Canonical timestamp and row-byte compatibility, including Unicode and Decimal representation
`test_synthetic_cli.py`	PostgreSQL dispatch and environment-based URL handling without printing credentials
`test_synthetic_postgresql_concurrency.py`	Bounded worker coordination and shutdown/failure behavior without a live database
`test_postgresql_synthetic_source.py`	Actual PostgreSQL concurrent loads, adapter compatibility, single-pass manifest equality, optional empty tables, artifacts, failed replacement preservation

Run focused checks through the repository's supported surface:

make test-focused TEST=tests/unit/test_synthetic_source.py
make test-focused TEST=tests/unit/test_synthetic_schedule.py
make test-focused TEST=tests/unit/test_synthetic_serialization.py
make docs-check

make test-postgres requires TEST_CONTROL_DATABASE_URL naming an explicitly disposable PostgreSQL database; these tests clean up generated tables. See testing for the broader verification ladder.

A schedule test at qualification size does not generate all 100 million views or exercise full training at that scale. Likewise, a reduced development benchmark does not establish Development Scale Confidence, and a successful PostgreSQL receipt does not establish a Qualification Claim. The latter requires the approved unmodified profile, environment, service build, and complete evidence gates. qualification_eligible only checks the generator-side input conditions; it is not a recorded qualification result.

Specifically, that flag requires the exact released qualification profile and profile version, the default root seed and fixed qualification cutoff (2026-01-01T00:00:00Z), both optional capabilities enabled, and empty overrides and calibration mappings. Nonempty calibration is rejected for a qualification-named configuration. Pass that cutoff explicitly for a qualification experiment; the current-time default does not qualify. Passing these input checks still does not run a qualification experiment.

The current bulk materializers are SQLite and PostgreSQL. Do not confuse the worker's internal DuckDB aggregate store with a generated-source DuckDB materializer, or treat the defined DuckDB Development Scale Confidence contract as implemented by a SQLite generation command.

16. Where to read or change the process¶

If you want to understand or change…	Start here	Keep in mind
Named workload sizes	`profiles.py`	Changing a released profile changes its contract
Configuration and reproducibility	`GenerationConfig`, `_substream_seed` in `generator.py`	Inputs, versions, and random-draw order matter
Catalog and cohort construction	`_build_catalog`	Metadata and evidence cohorts are separate dimensions
Session/Order shape	`_group_schedule`, `_base_group_size_sum`	Preserve exact quotas and feasible distinct group sizes
Product associations	`_products_for_group`, `_choose_product`	Add real source assertions and serving observations, not just signal names
Time and quantities	`_group_time`, `_quantity_for_row`	Preserve cutoff bounds, creation validity, and decimal precision
Expected outcomes	`_build_oracle_manifest`	An expectation still needs a separate evaluator
Content identity	`manifest.py`	Column order and canonical bytes are compatibility contracts
Local loading	`sqlite.py`	Two generation passes; direct canonical-table loading
Parallel loading and atomic publication	`postgresql.py`	Keep queues bounded and verification before publication
CLI behavior	`cli.py`	Credentials stay in environment values, not arguments or artifacts
Interactive activity generation	`simulation/scenario.py` and `simulation/runner.py`	Logical planning and runtime scheduling are separate

When extending the generator, first state the observable source property you need, then how the service should respond to it. For example, a real steady-versus-rising comparison needs explicit time-bucket counts and a corresponding ranking observation. Preserve reproducibility and scope, version intentional content changes, and verify at the smallest useful scale before attempting the full workload.