Development and benchmarks

Repository layout

Standard Rust checks

cargo fmt --all
cargo test --workspace
cargo clippy --workspace --all-targets

Before submitting a change, keep formatting, workspace tests, and Clippy across all targets clean. Changes to binary layouts should also run the core storage-format fixtures.

ANN benchmark

ann_bench compares IVF-FLAT, IVF-SQ, IVF-PQ, IVF-RQ, and DiskANN under one workload. By default it generates independent clustered queries and computes exact Top-10 ground truth. It can instead read standard fvecs/ivecs files and use their published ground truth. Every CSV row starts with a dataset label and reports vector and training counts, raw-dataset bytes, total build time plus train/add/write stages, process peak RSS, warm-up time, serialized bytes, Recall@10, first/P50/P95 latency, sequential and batch throughput, positional-read rounds, range count, and bytes.

cargo bench -p paimon-vindex-core --bench ann_bench -- --nocapture

ANN_N=100000 ANN_NQ=64 ANN_D=64 ANN_K=10 \
ANN_NLIST=256 ANN_NPROBE=32 ANN_PQ_CODE_RATIO=0.0625 \
ANN_DISKANN_L_SEARCH=200 \
ANN_RQ_BITS=4 \
cargo bench -p paimon-vindex-core --bench ann_bench -- --nocapture

ANN_DATA_GIB=10 ANN_D=1024 ANN_NOISE_DIMENSIONS=64 \
ANN_TRAIN_N=65536 ANN_NQ=8 ANN_K=10 \
ANN_NLIST=1 ANN_NPROBE=1 ANN_PQ_CODE_RATIO=0.0625 ANN_CLUSTERS=64 \
ANN_DISKANN_L_SEARCHES=200,1000 \
ANN_DISKANN_MEMORY_BUDGET_BYTES=34359738368 \
ANN_INDEXES=DISKANN ANN_KEEP_INDEXES=1 \
ANN_OUTPUT_DIR=/path/on/target/ssd \
cargo bench -p paimon-vindex-core --bench ann_bench

for index in IVF_FLAT IVF_SQ IVF_PQ IVF_RQ DISKANN; do
  ANN_DATA_GIB=10 ANN_D=1024 ANN_NOISE_DIMENSIONS=64 \
  ANN_TRAIN_N=65536 ANN_NQ=8 ANN_K=10 \
  ANN_NLIST=1024 ANN_NPROBE=64 ANN_PQ_CODE_RATIO=0.0625 ANN_CLUSTERS=64 \
    ANN_DISKANN_L_SEARCHES=200,1000 \
  ANN_DISKANN_MEMORY_BUDGET_BYTES=34359738368 \
  ANN_INDEXES="$index" ANN_OUTPUT_DIR=/path/on/target/ssd \
  cargo bench -p paimon-vindex-core --bench ann_bench
done

Public SIFT1M, GIST1M, and GloVe-100 data

ANN-Benchmarks publishes dense HDF5 files with independent test queries and exact Top-100 neighbors. Convert them once to the streaming-friendly fvecs/ivecs format. --query-limit=1000 keeps all recorded datasets at the same query count while retaining enough queries for stable Recall@10.

curl -LO https://ann-benchmarks.com/sift-128-euclidean.hdf5
python3 -m pip install h5py numpy
python3 tools/convert_ann_benchmarks.py \
  sift-128-euclidean.hdf5 /data/sift1m \
  --prefix sift1m --query-limit 1000

curl -LO https://ann-benchmarks.com/glove-100-angular.hdf5
python3 tools/convert_ann_benchmarks.py \
  glove-100-angular.hdf5 /data/glove100 \
  --prefix glove100 --query-limit 1000 --normalize-l2

RAYON_NUM_THREADS=12 ANN_DATASET_NAME=sift1m \
ANN_BASE_FVECS=/data/sift1m/sift1m_base.fvecs \
ANN_QUERY_FVECS=/data/sift1m/sift1m_query.fvecs \
ANN_GROUND_TRUTH_IVECS=/data/sift1m/sift1m_ground_truth.ivecs \
ANN_K=10 ANN_NLIST=1024 ANN_NPROBE=64 ANN_PQ_CODE_RATIO=0.0625 \
ANN_DISKANN_L_SEARCH=100 \
ANN_DISKANN_RAW_VECTOR_ENCODING=f16 \
ANN_DISKANN_MEMORY_BUDGET_BYTES=8589934592 \
ANN_STORAGE_CASES=local_ssd_warm_cache,remote_cache_2ms,object_store_20ms \
ANN_OUTPUT_DIR=/path/on/target/ssd \
cargo bench -p paimon-vindex-core --bench ann_bench

The benchmark infers ANN_N, ANN_NQ, and ANN_D from the public files, chooses ANN_TRAIN_N=min(N, max(65,536, 64 × nlist)), and automatically starts one child process per selected index so each peak-RSS value is independent. Explicit shape variables remain available as assertions. For GIST1M, download and convert gist-960-euclidean.hdf5; for GloVe-100, use the command above. Unit-normalized L2 has the same neighbor ordering as cosine for non-zero vectors; that equivalence was used by the recorded GloVe run. DiskANN now also accepts native cosine and inner-product metrics, which should be benchmarked against ground truth generated with the same production metric. Keep the other recorded settings unchanged. The default ANN_PQ_CODE_RATIO=0.0625 resolves automatically to m=32, m=240, and m=25 for SIFT, GIST, and GloVe respectively. Set ANN_PQ_M only to benchmark an explicit expert override.

At 1024 dimensions, ANN_DATA_GIB=10 derives exactly 2,621,440 vectors. Run large comparisons one index per process by changing ANN_INDEXES; this isolates process peak RSS and, unless ANN_KEEP_INDEXES=1, deletes that index after all selected serving cases. A comma-separated subset or all is convenient for smaller workloads. The benchmark retains only the selected training prefix, then adds the entire data set. The clustered generator is round-robin by cluster, so a prefix still covers every cluster when ANN_TRAIN_N ≥ ANN_CLUSTERS.

The benchmark executes three serving cases by default. local_ssd_warm_cache uses the real index file with no artificial delay. remote_cache_2ms adds a fixed 2 ms delay per pread call while reading all ranges in that call concurrently. object_store_20ms uses the ObjectStore read plan: open/optimization and sequential-query latency are the measured CPU/I/O time plus 20 ms for every observed dependent read round, avoiding hours of idle wall-clock sleep while retaining the exact fixed-latency model; batch QPS still injects the real 20 ms sleep so concurrent-client overlap is measured rather than inferred. These models do not include bandwidth or operating-system cache misses. DiskANN uses a dedicated range-I/O pool separate from Rayon query workers, preventing a saturated batch from starving nested positional reads; IVF retains the shared compute pool because its batch read scheduling is not nested inside per-query sessions. The unified IVF Reader reuses the 64-byte type-dispatch header and loads resident metadata in one further contiguous read, reducing initialization from three rounds to two without changing any format. DiskANN is opened explicitly as LocalStorage, RemoteStorage, or ObjectStore with the default 4 GiB Reader budget, which is automatically partitioned into resident state and caches. The sequential sweep and batch call use separate opened-and-optimized Readers, preventing all benchmark queries from prewarming their own batch measurement. CSV output records optimization rounds, ranges, and bytes separately from query I/O. Use ANN_STORAGE_CASES for a focused rerun against a retained index. See the recorded public-corpus results and interpretation.

DiskANN access-pattern benchmark

diskann_bench measures the page-oriented search path separately from the IVF benchmark. With no dataset paths it generates a small deterministic correctness workload. SIFT1M mode reads standard .fvecs and .ivecs files.

RAYON_NUM_THREADS=4 \
DISKANN_BENCH_ACCEPTANCE=1 DISKANN_BENCH_SIMULATED_RTT_MS=1 \
DISKANN_BENCH_CONCURRENCY=4 \
DISKANN_BENCH_STORAGE_LAYOUT=compact \
DISKANN_BENCH_RAW_VECTOR_ENCODING=f32 \
DISKANN_BENCH_BUILD_DISTANCE=product_quantized \
cargo bench -p paimon-vindex-core --bench diskann_bench

DISKANN_BASE_FVECS=/data/sift_base.fvecs \
DISKANN_QUERY_FVECS=/data/sift_query.fvecs \
DISKANN_GROUND_TRUTH_IVECS=/data/sift_groundtruth.ivecs \
DISKANN_BENCH_INDEX_DIR=/mnt/local-nvme \
RAYON_NUM_THREADS=16 \
DISKANN_BENCH_CONCURRENCY=16 \
DISKANN_BENCH_READER_MEMORY_BUDGET_BYTES=4294967296 \
DISKANN_BENCH_ACCEPTANCE=1 \
DISKANN_BENCH_FILTERED_MATRIX=1 \
DISKANN_BENCH_SIMULATED_RTT_MS=5 \
cargo bench -p paimon-vindex-core --bench diskann_bench

The benchmark writes a temporary .dann file under DISKANN_BENCH_INDEX_DIR (or the system temporary directory), reopens it through a benchmark-private SeekRead adapter, and deletes it on exit. This adapter exercises native positional I/O without adding a local-file type to the public core API. Use DISKANN_BENCH_STORAGE_LAYOUT=compact|interleaved, DISKANN_BENCH_RAW_VECTOR_ENCODING=f32|f16, and DISKANN_BENCH_BUILD_DISTANCE=full_precision|product_quantized for controlled comparisons. PQ defaults to DISKANN_BENCH_PQ_CODE_RATIO=0.0625; DISKANN_BENCH_PQ_M is an optional explicit override in 1..=dimension and DISKANN_BENCH_PQ_BITS selects 4 or 8 bits. Generated workloads can be scaled with DISKANN_BENCH_SMOKE_N, DISKANN_BENCH_SMOKE_NQ, and DISKANN_BENCH_SMOKE_DIMENSION; exact top-10 truth is generated automatically. CSV rows inject representative latency hints for all four internal read tiers at l_search 50, 100, and 200 so timing noise does not make planner comparisons ambiguous. DISKANN_BENCH_SIMULATED_RTT_MS adds the same fixed delay to every pread round for every tier. Set DISKANN_BENCH_WARMUP_QUERIES=0 to measure resident-only initialization; its default 8 replays representative queries before latency sampling. DISKANN_BENCH_READER_MEMORY_BUDGET_BYTES controls the one public Reader budget; the benchmark records automatically resolved cache behavior through search statistics. Rows additionally identify layout, raw-vector encoding, build distance, selected graph-shard count, and complete parallel-session query counts.

Set DISKANN_BENCH_FILTERED_MATRIX=1 to add random and contiguous/cluster-correlated filters at 0.01%, 0.1%, 1%, 10%, 50%, and 100% selectivity. The matrix compares adaptive results with a forced exhaustive matching-node baseline. It reports the actual executed strategy and graph fallback count from diskann_search_stats, Recall@10, elapsed time, I/O, PQ distance evaluations versus PQ code loads, query chunks, rerank chunks, candidate references, unique raw windows, batch LRU hits/misses/evictions, and parallel exact-rerank work. Under DISKANN_BENCH_ACCEPTANCE=1, every graph-enabled cell must stay within one absolute Recall@10 percentage point of the exhaustive baseline.

The benchmark warms one Reader. DISKANN_BENCH_CONCURRENCY=1 repeatedly invokes the single-query API; larger values invoke the native batch path at that width. Retained sessions run complete queries concurrently for small batches and all interleaved-layout batches. Large compact batches retain shared vector-window rerank. Graph sessions clone only the storage handle and share resident PQ/row-ID/code allocations, validation state, hot prefix, and cold-adjacency LRU. Local results use the operating-system page cache after index creation; set DISKANN_BENCH_INDEX_DIR to the production SSD mount and run cold-cache testing with an external, platform-appropriate cache-control procedure.

C FFI smoke test

Build the release shared library, then compile the C test against the generated header.

cargo build --release -p paimon-vindex-ffi
cmake -S c -B c/build
cmake --build c/build
LD_LIBRARY_PATH=target/release c/build/test_vindex

C++ smoke test

The C++ test reuses the same FFI shared library and validates the RAII header.

cargo build --release -p paimon-vindex-ffi
cmake -S cpp -B cpp/build
cmake --build cpp/build
LD_LIBRARY_PATH=target/release cpp/build/test_vindex_cpp

Java / JNI tests

The Maven test phase compiles the Java sources and runs VectorIndexJavaApiTest. This test covers result and metadata objects, native resource mapping, closed-handle behavior, and API compilation without loading the native library:

mvn -f java/pom.xml test

Native validation, panic-boundary, and handle-safety tests are standalone main programs. The following Linux commands mirror the additional JNI checks run by CI:

cargo build -p paimon-vindex-jni --release

java -cp java/target/test-classes:java/target/classes \
  org.apache.paimon.index.vector.VectorIndexNativeValidationTest \
  "$(pwd)/target/release/libpaimon_vindex_jni.so"
java -cp java/target/test-classes:java/target/classes \
  org.apache.paimon.index.vector.VectorIndexNativePanicBoundaryTest \
  "$(pwd)/target/release/libpaimon_vindex_jni.so"
java -cp java/target/test-classes:java/target/classes \
  org.apache.paimon.index.vector.VectorIndexNativeHandleSafetyTest \
  "$(pwd)/target/release/libpaimon_vindex_jni.so"

Release CI additionally builds the multi-platform Maven JAR and runs VectorIndexNativeLoaderSmokeTest from the final java-package artifact with no native path argument on Linux x86-64, Linux aarch64, macOS arm64, and Windows x86-64. This ensures consumers can use every bundled JNI library directly.

Python ctypes tests

The Python package must be able to locate the newly built FFI library:

cargo build --release -p paimon-vindex-ffi
cd python
PAIMON_VINDEX_LIB_PATH=../target/release pip install -e ".[test]"
PAIMON_VINDEX_LIB_PATH=../target/release pytest -v

The suite uses NumPy inputs and covers all index types, persisted RQ bit metadata, single/batch shapes, and error paths.

Storage format and compatibility

Changes to headers, flags, sections, sort order, or graph encoding must update both format fixtures and documentation. The normative v1 field definitions live in core/STORAGE_FORMAT.md.