Vec3 Comparison
Chroma vs pgvector benchmark suite
About
End-to-end benchmarking pipeline comparing Chroma and pgvector across 10 datasets ranging from 10K to 2M vectors and dimensions from 32 to 1536. Measures throughput, latency percentiles (mean, P50, P99), recall@k, and resource usage; supports HNSW and IVFFlat indexes and multiple batch sizes. Containers are resource-capped (4 GB memory, 6 CPU cores) for fair comparison. React dashboard visualizes results, with JSON, CSV, plot, and LaTeX-table export. Built as an NTUA university project; the accompanying paper found pgvector roughly 2-5x faster on ingestion and lower-latency on queries, while Chroma kept consistently higher recall@k.
Status
Benchmarks ran across the full dataset range. Paper is published as part of the NTUA submission.