Embedded ANN native index experiment¶
Status: development evidence only, 2026-09-24. This is a synthetic Faiss index comparison for Benchmark embedded ANN on Apple Silicon. The other ticket criteria, especially the approved Item representation, budgets, multi-process resource evidence and supported-platform matrix, remain open.
Reproduce¶
make benchmark-ann ANN_BENCHMARK_ARGS='--items 2000 --queries 40'
make benchmark-ann ANN_BENCHMARK_ARGS='--items 200000 --queries 100 --ef-search 64'
make benchmark-ann ANN_BENCHMARK_ARGS='--items 200000 --queries 100 --ef-search 256'
make benchmark-ann ANN_BENCHMARK_ARGS='--items 200000 --queries 100 --ef-search 256 --eligible-fraction 0.1'
make benchmark-ann ANN_BENCHMARK_ARGS='--items 200000 --queries 100 --ef-search 256 --concurrent-workers 4'
make benchmark-ann ANN_BENCHMARK_ARGS='--items 200000 --queries 100 --ef-search 256 --eligible-fraction 0.001 --concurrent-workers 4'
make benchmark-ann ANN_BENCHMARK_ARGS='--items 200000 --queries 100 --ef-search 512 --eligible-fraction 0.001 --concurrent-workers 4'
The command adds pinned faiss-cpu==1.15.1 only to its experiment process. It generates
fixed-seed, normalized, clustered 64-dimensional float32 vectors. Half the queries use
one eligible Item vector with its own ID excluded; half average four Item vectors to
approximate the shape of a Shopper query. The profile-shaped inputs can include
Items that are currently ineligible, representing historical signals only. The
eligibility mask is applied at query time over a full synthetic index; this
stress-tests restrictive filters and does not claim that a production index should
retain every ineligible Catalog Item. Both Flat and HNSW search the same vectors
with native eligibility selectors. Reported recall compares HNSW with exact Flat
on the same eligible
population. It excludes queries for which the exact eligible set is empty rather than
scoring those as perfect recall. There are no merchant interactions or Shopper identities.
The command checks the first three Flat results against independent NumPy dot-product
ranking (score-equivalent ties are accepted), rejects duplicate, ineligible or anchor
IDs, and verifies the first HNSW query after serialization and warm in-process reload.
It writes the same serialized index to a bounded temporary directory, starts a new
Python process, imports Faiss, loads the index, checks its Item count and performs
one search. The temporary artifact is removed after the child exits or times out.
The process wall time includes interpreter startup, imports and file load; the child
reports Faiss import time, read_index time and its own peak RSS. Because the parent
just wrote the file, filesystem cache may make this faster than a cold-disk read.
The optional --concurrent-workers setting (1–8, no more than the query count) runs
the same filtered HNSW queries through a bounded CPU thread pool, checks that every
result matches sequential search, and reports native-call latency and whole-pool wall
throughput separately. This read-only pattern follows Faiss's
CPU threading guidance;
it does not imply that the Metal GPU index is safe for shared concurrent use.
Local results¶
Native M2 Pro, 32 GiB RAM, macOS 27.0 ARM64, Python 3.14.7, NumPy 2.5.2, Faiss 1.15.1;
2 Faiss threads, HNSW 16 links and construction ef 80. The installed wheel reported
ARM_NEON and MAC_METAL compile options. The measured indexes were ordinary CPU
IndexFlatIP and IndexHNSWFlat; these measured searches did not use Metal.
The same installed wheel returned one device from get_num_gpus() and successfully
searched an eight-Item Flat index after index_cpu_to_gpu. A second check supplied a
native selector permitting only IDs 1, 2 and 3, but the GPU result included ID 0.
Treat that GPU bridge as unsuitable for eligible-candidate serving until filtering is
implemented and verified at a proper seam. The upstream source includes a Metal
build option
and a Python bridge;
the runtime check establishes that this wheel exposes it. It does not establish a
Metal speed advantage or Neural Engine support.
An additional one-off, unfiltered Flat comparison used 200,000 synthetic vectors at 256 dimensions, 100 sequential one-query searches, two CPU threads and one warmup. CPU Flat p95 was 3.60 ms; the Metal bridge p95 was 5.41 ms. This is not the filtered workload below, was not repeated, and did not include concurrent or batched queries. It gives no basis to substitute the Metal bridge for eligible-candidate retrieval.
| Items | Dimensions | Eligible fraction | Search ef | HNSW build | Exact p95 | HNSW p95 | Recall@10 | Fill | Peak process RSS | Serialized index |
|---|---|---|---|---|---|---|---|---|---|---|
| 2,000 | 64 | 0.5 | 64 | 0.054 s | 0.026 ms | 0.033 ms | 0.995 | 1.00 | 60 MiB | 0.8 MB |
| 200,000 | 64 | 0.5 | 64 | 6.80 s | 2.49 ms | 0.154 ms | 0.659 | 1.00 | 631 MiB | 80.1 MB |
| 200,000 | 64 | 0.5 | 256 | 6.76 s | 2.52 ms | 0.476 ms | 0.807 | 1.00 | 631 MiB | 80.1 MB |
| 200,000 | 64 | 0.1 | 256 | 6.77 s | 0.889 ms | 0.372 ms | 0.792 | 1.00 | 603 MiB | 80.1 MB |
| 200,000 | 256 | 0.5 | 256 | 22.07 s | 5.65 ms | 0.904 ms | 0.698 | 1.00 | 1.61 GiB | 233.7 MB |
Values are single-run observations, rounded from command output. p95 is per-query
native index time for 40 or 100 queries after one warmup. The high-water RSS includes
both original indexes, serialized bytes and the reloaded index, not just HNSW. The
command also reports a separate pre-reload process peak. Other concurrently running
work, additional Commerce Scopes, snapshot generations and application workers were
not measured. No confidence interval or production speed claim follows.
For the 200,000-Item, 64-dimensional, ef 256 row, warm in-process deserialization took 0.010 s and sequential HNSW native-call throughput was 2,872 queries/s. These throughput figures sum sequential per-query native-call durations; they do not model complete API throughput, encoding or filtering after retrieval.
A separate single-run 200,000-Item, 64-dimensional, ef 256 experiment with four concurrent workers and two configured Faiss threads per worker returned the same filtered IDs as sequential search. At 50% eligibility, concurrent native-call p95 was 0.548 ms and whole-pool wall throughput was 9,589 queries/s over 100 queries; at 10% eligibility, p95 was 0.444 ms and throughput 10,691 queries/s. Recall@10 against exact Flat was 0.807 and 0.792 respectively, with full top-ten fill. These are short synthetic bursts in one process, including pool startup in the wall rate but excluding Serving API work, concurrent scope generations and sustained load. They are not capacity limits or an approval of four workers.
After adding the fresh-process load check, another single 200,000-Item,
64-dimensional, 50%-eligible, ef 256 run loaded the 80.1 MB serialized HNSW index
in a new process: Faiss import 0.020 s, read_index 0.018 s, process wall time
0.102 s, child peak RSS 130 MB. The child searched one unfiltered vector and
verified 200,000 indexed Items; the separate in-process reload check exercised a
filtered query. These one-run numbers exclude full service startup, other Commerce
Scopes, overlapping snapshot generations and genuine cold-disk conditions.
Selective eligibility stress¶
The same 200,000-Item, 64-dimensional synthetic setup was run with much smaller
eligible fractions to expose filtered-search underfill. Exact Flat searched the
same eligible population and excluded each Similar Items anchor. Underfilled
means HNSW returned fewer IDs than Flat for that query; it is distinct from low
recall among returned IDs.
| Eligible fraction | Eligible Items | Search ef | Flat fill@10 | HNSW fill@10 | Queries underfilled vs Flat | Recall@10 | Flat p95 | HNSW p95 |
|---|---|---|---|---|---|---|---|---|
| 0.01 | 2,007 | 256 | 1.00 | 1.00 | 0% | 0.707 | 0.552 ms | 0.356 ms |
| 0.001 | 200 | 256 | 1.00 | 0.327 | 100% | 0.256 | 0.476 ms | 0.380 ms |
| 0.001 | 200 | 512 | 1.00 | 0.502 | 91% | 0.393 | 0.483 ms | 0.852 ms |
At 0.1% eligibility, raising ef improved fill but did not remove underfill; in that run exact Flat was faster than HNSW at ef 512 while filling every quota. This is evidence against treating filtered HNSW as a complete source of eligible candidates at every selectivity. An exact search of the eligible subset, or a verified bounded fallback when HNSW underfills, is a candidate for the index and publication decisions. These synthetic single runs do not set a production selectivity threshold, acceptable recall, or fallback latency budget.
The 256-dimensional row was run separately with --dimensions 256 --items 200000
--queries 100 --ef-search 256. Its synthetic vectors and eligible set differ from the
64-dimensional rows under the same seed, so the recall values are not a controlled
dimension-only comparison. Its much larger process RSS shows why the index and vector
dimension choices need an explicit multi-scope memory budget.
The low 200,000-Item recall at ef 64, and its improvement at ef 256, make recall and resource budgets necessary before index selection. These synthetic vectors are not the current sparse Item Metadata representation, and the queries are not actual authorized Shopper Profiles. This experiment does not measure recommendation relevance, full Serving API latency, Linux performance, service restart, artifact publication or formal qualification.
Metadata representation comparison¶
make benchmark-ann-representation uses the committed smoke Generated Relational Data
Source's Catalog. It calls the existing Similar Items builder for exact sparse top-ten
reference results, then fits deterministic truncated SVD projections of the same
metadata features. It compares each dense vector's exact cosine neighbors with those
reference results. This isolates representation loss from ANN approximation error.
The smoke Catalog yielded 903 eligible Items with metadata, 21,218 sparse features, and ten reference neighbors for each anchor. All projections below were fitted and compared in this local process; times exclude source generation and sparse baseline construction.
| Dense dimensions | Exact dense neighbor overlap with current sparse top ten | Fit and compare | Dense vector bytes |
|---|---|---|---|
| 64 | 0.682 | 0.113 s | 231,168 |
| 128 | 0.739 | 0.200 s | 462,336 |
| 256 | 0.797 | 0.369 s | 924,672 |
| 512 | 0.851 | 0.765 s | 1,849,344 |
These overlaps are not relevance judgments. The generated Catalog has repeated titles, brands and categories, so other Catalogs may differ substantially. At 200,000 Items, the vector matrix alone would occupy 204.8 MB at 256 float32 dimensions, before an index, transform, Product ID mapping, workers or overlapping snapshot generations. The representation decision remains open and must consider both Similar Items and For You query construction.
Verification performed¶
- Pinned Faiss wheel imported on the native host and exposed Flat, HNSW and ID selectors.
- Experiment tests: 14 passed with the optional wheel installed.
- Representation experiment tests: 2 passed; the smoke command above completed.
make test-static: passed.make test: 499 unit passed, 1 optional Faiss test skipped in the locked base environment, 42 contract/delivery passed, 33 non-PostgreSQL integration passed; 20 PostgreSQL tests deselected because no disposableTEST_CONTROL_DATABASE_URLwas configured.git diff --check: passed.
Pinned-wheel compatibility smokes¶
PyPI's Faiss 1.15.1 release metadata
publishes the following SHA-256 digests for the glibc-based wheels relevant to
these checks. Streaming each corresponding download from files.pythonhosted.org
through shasum -a 256 produced the same digest. This verifies the three
downloaded artifacts against PyPI's published values, not the separate ephemeral
wheel bytes installed by uv run; the latter has not been independently captured.
PyPI metadata identifies Meta AI Research as author, links to the
Facebook Research Faiss repository and declares MIT; the upstream tagged
license file
also states MIT. Package provenance and transitive native-library obligations
still require review before a production dependency pin.
| Wheel platform | Published filename | Published SHA-256 |
|---|---|---|
| macOS ARM64 | faiss_cpu-1.15.1-cp310-abi3-macosx_14_0_arm64.whl |
ea9e12d540ca8ac0347b831d034c0f6d7ff5eed20523a247db44b3543ad2aad4 |
| Linux ARM64 | faiss_cpu-1.15.1-cp310-abi3-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl |
ffa71b14b3090bc076f8b026554178868fdbfe2f26fe644da629405836369039 |
| Linux x86_64 | faiss_cpu-1.15.1-cp310-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl |
f2c31b7f2f6647eb76829a5cfe3c398fb9346df9f26b1d4db35269c91eb58c33 |
The published macOS ARM64 wheel tag requires macOS 14 or newer; the Linux wheels carry manylinux glibc 2.27/2.28 tags. This repository does not yet declare an explicit minimum macOS version. Resolve that support-policy choice before pinning Faiss rather than silently excluding older Apple Silicon hosts.
Each row installed faiss-cpu==1.15.1 into a disposable uv run --no-project
environment (or a disposable Linux container), imported Faiss and NumPy, built a
four-dimensional Flat index and returned its expected nearest Item. These are wheel
and native-load checks; they do not install the full recommendation distribution or
prove artifact portability or serving performance.
| Host | Python | Architecture | NumPy | Result |
|---|---|---|---|---|
| macOS 27 M2 Pro | 3.12.10 | arm64 | 2.5.3 | Passed |
| macOS 27 M2 Pro | 3.13.15 | arm64 | 2.5.3 | Passed |
| macOS 27 M2 Pro | 3.14.7 | arm64 | 2.5.2 | Passed |
| Linux Bookworm container | 3.12.12 | aarch64 | 2.5.3 | Passed |
| Linux Bookworm container | 3.13.12 | aarch64 | 2.5.3 | Passed |
| Linux Bookworm container | 3.14.2 | aarch64 | 2.5.3 | Passed |
| Linux Bookworm container on M2 Pro | 3.12.12 | x86_64 emulated | 2.5.3 | Passed under emulation |
| Linux Bookworm container on M2 Pro | 3.13.12 | x86_64 emulated | 2.5.3 | Passed under emulation |
| Linux Bookworm container on M2 Pro | 3.14.2 | x86_64 emulated | 2.5.3 | Passed under emulation |
The disposable Linux containers used the
ghcr.io/astral-sh/uv:python3.14-bookworm-slim image and uv-managed Python
3.12/3.13 interpreters, then were removed. The locally inspected ARM64 image
digest was
sha256:7cf77f594be8042dab6daa9fe326f90962252268b4f120a7f5dccce4d947e6c1;
the x86_64 image digest was not recorded. The emulated x86_64 results are packaging
smokes, not native x86 performance evidence. The
macOS wheel reported ARM_NEON MAC_METAL, Linux aarch64 reported ARM_NEON, and
Linux x86_64 reported AVX2 in their compile options. The pinned Faiss wheel now has
an import-and-search smoke across Python 3.12–3.14 and these three host/architecture
settings. Additional Python 3.14 Linux ARM64 and emulated x86_64 smokes built HNSW
indexes and confirmed native eligibility selectors excluded an ineligible nearest
Item. These tiny checks do not prove full workload parity, and no clean installed
recommendation distribution, native Linux x86_64
performance or sustained Linux concurrency has been run.
A separate tiny portability smoke serialized a five-Item, four-dimensional HNSW
index with Faiss 1.15.1 on the native macOS ARM64 host. Disposable Linux ARM64 and
emulated Linux x86_64 containers read the same 798-byte artifact with the same
Faiss version and returned Item 1 for a filtered top-one query excluding the
unfiltered nearest Item 0. The experimental raw Faiss artifact SHA-256 was
8ee1aa225d4135c1fddfe71dd0c2194b82f222a1bb8d339d521732f677811d6e;
it was removed after the check. This establishes only small same-version
serialization compatibility across those three environments. It does not prove
large-index portability, cross-version compatibility, integrity verification at
publication, or behavior on native Linux x86_64 hardware.
All tabled native-index runs were repeated after the eligible-anchor correction, empty-reference recall guard and reload check. The older numbers posted to Linear before that correction are superseded. Each corrected row is still a single run; repeat measurements before using any row as a qualification gate.