A new study on arXiv evaluates how hierarchical text embedding models and quantization techniques interact, finding that 8-bit integer quantization performs comparably to 32-bit floating-point precision for certain tasks. The research analyzed 168 configurations across 38 tasks from the Massive Text Embedding Benchmark (MTEB).
The study, titled "Balancing hierarchical representation and quantization: A performance trade-off in text embeddings," investigated the efficiency and performance implications of combining hierarchical text representations with different levels of quantization. Researchers found that 8-bit integer (int8) quantization is statistically indistinguishable from standard 32-bit floating-point (float32) for pair classification and re-ranking tasks. This level of quantization resulted in a minimal performance cost of at most 1.5 percentage points overall, while significantly reducing storage requirements. Binary quantization, however, showed a substantial performance decrease across all evaluated task categories.
The research also explored the impact of reducing embedding dimensionality. Halving the embedding dimension once did not yield statistically significant performance changes for dimensions above 16. However, two successive halvings consistently led to significant performance drops. Combining an eight-fold dimensionality reduction with int8 quantization achieved a 32-fold reduction in per-vector storage while retaining 94.4% of the original effectiveness.
Regarding evaluation speed, the study noted that quantization itself did not provide evidence of acceleration. Instead, sampling from intermediate layers of the embedding models proved to be a more effective method for reducing evaluation time. The performance degradation was most pronounced in retrieval tasks, which degraded faster than other categories.
The Massive Text Embedding Benchmark (MTEB) is a widely used resource for evaluating text embedding models, encompassing 8 distinct task types and 58 datasets across 112 languages. The benchmark aims to provide a comprehensive assessment of embedding model capabilities beyond single tasks like semantic textual similarity, which has historically dominated evaluations. The study utilized four transformer-based embedding models and examined both one-dimensional (1D) and two-dimensional (2D) hierarchical representations.
The findings suggest that practitioners can achieve substantial gains in storage efficiency by employing int8 quantization and dimensionality reduction without a significant compromise in performance for many common text embedding tasks. The research highlights the trade-offs between representation complexity, numerical precision, and computational cost, offering practical guidance for optimizing text embedding models.