Implement filtered diskann (#205)
# Filtered DiskANN: Label-Based Vector Search Filtering
This PR implements label-based filtering for vector searches in
pgvectorscale, allowing users to efficiently filter vector search
results based on integer labels associated with each vector. This
feature significantly improves performance compared to post-filtering
approaches.
## Overview
The implementation adds support for filtering vector search results
using the PostgreSQL array overlap operator (`&&`), which allows for
efficient filtering of vectors based on their associated labels. This is
particularly useful for applications that need to search within specific
categories, domains, or other classification systems.
For detailed information on the design and implementation, see the
[Filtered DiskANN Design
Document](https://timescale.slab.com/posts/filtered-disk-ann-for-pgvectorscale-aptwfj2g?shr=0XYDEW5mSRdgk8sFfEaYDvcn).
## Key Features
- **Label-Based Filtering**: Filter vector search results based on
integer labels stored in an array column
- **Performance Optimization**: Filtering happens during the graph
traversal, not as a post-processing step
- **Semantic Label Support**: Documentation on how to map integer labels
to meaningful semantic categories
- **Comprehensive Test Coverage**: Added test cases for various label
filtering scenarios
## Usage Example
```sql
-- Create a table with vectors and labels
CREATE TABLE documents (
id SERIAL PRIMARY KEY,
embedding VECTOR(384),
labels SMALLINT[],
content TEXT
);
-- Create a DiskANN index on the embedding column with label support
CREATE INDEX idx_documents_embedding
ON documents
USING diskann(embedding, labels);
-- Perform label-filtered vector searches
SELECT * FROM documents
WHERE labels && ARRAY[1, 3]::SMALLINT[] -- Documents with label 1 OR 3
ORDER BY embedding <=> '[...]'
LIMIT 10;
---------
Signed-off-by: tjgreen42 <tj@timescale.com>
Co-authored-by: Matvey Arye <cevian@gmail.com> T
tjgreen42 committed
cec0e86d2433ad8fe24e8bc3d23ea179a0f280c1
Parent: 52988c6
Committed by GitHub <noreply@github.com>
on 3/19/2025, 10:41:35 PM