VecForge research data

Experiments and scripts for VecForge 2.0

Results, benchmark numbers and scripts for the VecForge 2.0 paper. VecForge is an embedded, local-first vector database for Python. It combines hybrid search, encrypted storage and persistent memory for AI assistants over the Model Context Protocol, and each feature is tested in 18 experiments.

Author Suneel Bose K, Founder and CEO, ArcGX TechLabs Private Limited, Tiruchirappalli
Status Under consideration, 2026
Software VecForge 2.0.0

VecForge on PyPI ORCID Downloads

Key results

All numbers come from logged runs. The logs are in the results download.

0.707nDCG@10 for hybrid search on SciFact, against 0.645 for dense search alone (300 development queries)

0.0002Significance (paired randomisation test) of hybrid over dense search on the same queries

4.3 msMedian hybrid search over 5,183 SciFact documents, on a 4-core virtual machine

18Experiments covering every feature, with negative results reported alongside the positive ones

Retrieval quality on SciFact

SciFact from the BEIR benchmark, with the all-MiniLM-L6-v2 encoder. The fusion weight α was tuned on 809 training queries, where the best value was 0.5, and the table reports the 300 development queries.

Method	nDCG@10	95% interval	Recall@100	MRR@10
BM25 only	0.665	0.618 to 0.710	0.886	0.634
Dense only (MiniLM)	0.645	0.598 to 0.690	0.925	0.605
Reciprocal rank fusion, k = 60	0.681	0.637 to 0.724	0.954	0.639
Hybrid, α = 0.5	0.707	0.663 to 0.749	0.962	0.672

Downloads

Each download is a zip file with a short readme inside.

Experiment results 0.02 MB

results.json (E1 to E7) and results2.json (E8 to E18), with the environment of each run. The readme maps every experiment to its measurements.

Experiment and benchmark scripts 0.05 MB

The scripts that run and plot every experiment, the search and reranking benchmarks, the example applications and two quick checks of the scoring.

How to reproduce

Install the package: pip install "vecforge[mcp]".
Run E1 to E7 with python run_experiments.py. This takes about 15 minutes on 4 cores.
Download SciFact and the all-MiniLM-L6-v2 encoder, set SCIFACT_DIR and MINILM_DIR, then run E8 to E18 with python run_experiments2.py. Use --only E9,E10 to rerun a subset.
Draw the figures with make_figures.py and make_figures2.py.

The experiments also need onnxruntime, tokenizers and the mcp client package. Timings depend on the machine, so compare ratios rather than absolute times.

Data sources

SciFact: Wadden et al. (2020), doi.org/10.18653/v1/2020.emnlp-main.609, distributed in the BEIR benchmark (Thakur et al., 2021), github.com/beir-cellar/beir.
Encoder: all-MiniLM-L6-v2, a sentence encoder distilled from MiniLM (Wang et al., 2020).
Full source code: github.com/bosekarmegam/vecforge, under the Business Source License 1.1.

The SciFact dataset and the encoder are not included here. Download them from their sources and check their licences. The paper is under consideration and is not published. This page covers the data and scripts only.

How to cite

Suneel Bose K. (2026). VecForge: A Local-First Hybrid Vector Database with Persistent Agent Memory and Order-Preserving Grover-Inspired Reranking [Manuscript under consideration]. ArcGX TechLabs Private Limited. https://www.bosekarmegam.in/research/vecforge/