Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
345b9d1
MOD-14916 Devirtualize HNSW / brute-force search hot path (#937)
ofiryanai Apr 19, 2026
4f9d216
initial
Jun 23, 2026
b5c1c08
optimization
Jun 26, 2026
417b970
fix threadpool
Jun 26, 2026
2d5d699
minor fix
Jun 26, 2026
3c3f842
minor cleaning
Jun 29, 2026
952efbf
clean up
Jul 1, 2026
8265a7a
remove dead code
Jul 1, 2026
71c92bd
fixes; tests pass
Jul 8, 2026
b130e31
refactoring
Jul 8, 2026
78b4ef0
fixes for GC
Jul 15, 2026
fedd22e
copy blob before flat unlock
Jul 20, 2026
fcf83dd
Remove null characters from end of file
mihaic Jul 24, 2026
7b71aac
Merge pull request #1 from mihaic/lockfree-fix-end
razdoburdin Jul 25, 2026
4035699
fixes for gc
Jul 28, 2026
5f182bb
Merge branch 'lockfree' of https://github.com/razdoburdin/VectorSimil…
Jul 28, 2026
649460a
fix writer starvation for compaction
Jul 30, 2026
07f2c8f
adopt sigle label consolidation
Aug 11, 2026
db1d106
vecsim separate fin-grain index enabling
ethanglaser Aug 22, 2026
a540c0a
Rename SVSIndexBase thread API to match upstream
ethanglaser Aug 27, 2026
29c6af6
Merge upstream/main into dev/eglaser-lockfree
ethanglaser Aug 27, 2026
ac97868
CI fixes
ethanglaser Sep 2, 2026
a32aa9d
clean-up commented and unused dead code
Sep 3, 2026
978a49e
harmonize test
Sep 3, 2026
30abed6
Merge branch 'main' into dev/eglaser-lockfree
razdoburdin Sep 4, 2026
2a06ff7
minor revisions for CI
ethanglaser Sep 4, 2026
095ae3b
fix build and tests
Sep 7, 2026
6379bfd
Merge branch 'dev/eglaser-lockfree' of https://github.com/ethanglaser…
Sep 7, 2026
cce885c
fix ub
Sep 8, 2026
f13bbb5
use SVS rc
ethanglaser Sep 9, 2026
eea54c5
fix TestIndexMultiValueJsonReload
razdoburdin Sep 10, 2026
7376b1f
clang format
Sep 10, 2026
a46de7f
cland-format-18
Sep 10, 2026
ffb1e84
num_marked_deleted tracks not compacted entries now
Sep 10, 2026
5bf39ab
clean-up and fix
Sep 11, 2026
b4a2d83
update SVS submodule reference
ethanglaser Sep 11, 2026
87d65f3
Merge RedisAI/main into dev/eglaser-lockfree
ethanglaser Sep 11, 2026
a544c4c
Fix semantic fallout from the main merge
ethanglaser Sep 11, 2026
3e0a2cf
Satisfy check-format
ethanglaser Sep 11, 2026
fb1f99e
improve svs pimpl guarding for concurent execution
Sep 14, 2026
cc0d3db
resolve conflicts
Sep 14, 2026
f301d15
format
Sep 14, 2026
93ad60c
fix leaking consolidate jobs
Sep 14, 2026
f34be24
resubmit insert job if backend isn't ready
Sep 14, 2026
86feaa6
use RAII lock to harmonize topKQuery and rangeQuery
ahuber21 Sep 14, 2026
2ff7cf2
fix and optimize
Sep 14, 2026
ded890c
generalize SharedMutexLockable and harominze SVS/HNSW functions with …
ahuber21 Sep 14, 2026
468e6ef
move droping of gc schdule flag
Sep 14, 2026
9050eb2
cleanup comments
ahuber21 Sep 14, 2026
bb11940
fixup
ahuber21 Sep 14, 2026
c0d3a96
fix comment formatting
ahuber21 Sep 14, 2026
a42c4eb
Harmonize lockfree SVS and HNSW
ethanglaser Sep 14, 2026
ab6bfde
add missing file
ahuber21 Sep 14, 2026
cc9b666
use SVS rc2 binaries and latest submodule
ethanglaser Sep 15, 2026
86a26f1
fix iterator
Sep 15, 2026
8439ff7
Merge branch 'dev/eglaser-lockfree' of https://github.com/ethanglaser…
Sep 15, 2026
c1797a8
Merge upstream/main into dev/eglaser-lockfree
Sep 15, 2026
964a913
relabelVector: switch check to concurent; rewrite impl call logic
Sep 15, 2026
708e285
Merge branch 'main' into dev/eglaser-lockfree
razdoburdin Sep 15, 2026
aedb77c
harmonize svs and hnsw tiered job types
ahuber21 Sep 15, 2026
0b68c82
Revert "harmonize svs and hnsw tiered job types"
ahuber21 Sep 15, 2026
486cbaa
Merge remote-tracking branch 'upstream/main' into dev/eglaser-lockfree
ahuber21 Sep 15, 2026
90b22f8
adopt relabelVector for concuren svs
Sep 15, 2026
1082467
Merge branch 'dev/eglaser-lockfree' of https://github.com/ethanglaser…
Sep 15, 2026
ffab818
move hnsw/svs tierd common logic to the base class
Sep 16, 2026
b48755c
format
Sep 16, 2026
9e602e9
upstream getDistanceFrom_Unsafe
ahuber21 Sep 16, 2026
c5c2418
fix Full-buffer overwrite skips pending jobs
Sep 16, 2026
b2873d8
fix Init overwrite reports a new insert
Sep 16, 2026
2089fed
fix Empty index skips compression retraining
Sep 16, 2026
966067c
update comment for GC compacts without consolidating deletes
Sep 16, 2026
fc2a99a
Merge branch 'dev/eglaser-lockfree' of https://github.com/ethanglaser…
Sep 16, 2026
6e19eb6
fix Insert job can reinit dropped backend
Sep 16, 2026
6479bd3
modify comments
Sep 16, 2026
45005b6
fix Relabel races pending multi-value ingest
Sep 16, 2026
f5ab8da
bump glibc version error message to 2.28
ethanglaser Sep 16, 2026
b532111
format
Sep 16, 2026
30fbecb
Merge branch 'dev/eglaser-lockfree' of https://github.com/ethanglaser…
Sep 16, 2026
ca35f7c
fix Relabel duplicates in-flight ingested vectors
Sep 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 5 additions & 7 deletions cmake/svs.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ if(USE_SVS)
if (CMAKE_CXX_COMPILER_ID MATCHES "Clang")
if(GLIBC_VERSION VERSION_GREATER_EQUAL "2.31" AND CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL "21.0")
if(GLIBCXX_VERSION VERSION_GREATER_EQUAL "11")
set(SVS_URL "https://github.com/intel/ScalableVectorSearch/releases/download/v0.3.2/svs-shared-library-reduced-clang21-gcc11.tar.gz" CACHE STRING "SVS URL")
set(SVS_URL "https://github.com/intel/ScalableVectorSearch/releases/download/nightly/svs-shared-library-0.5.0rc2-reduced-clang21-gcc11.tar.gz" CACHE STRING "SVS URL")
else()
message(STATUS "libstdc++ >= GCC 11 is required for Clang SVS binaries - disabling SVS_SHARED_LIB")
set(SVS_SHARED_LIB OFF)
Expand All @@ -76,14 +76,12 @@ if(USE_SVS)
else()
if(GLIBC_VERSION VERSION_GREATER_EQUAL "2.28")
if(CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL "14.0")
set(SVS_URL "https://github.com/intel/ScalableVectorSearch/releases/download/v0.3.2/svs-shared-library-reduced-gcc14.tar.gz" CACHE STRING "SVS URL")
set(SVS_URL "https://github.com/intel/ScalableVectorSearch/releases/download/nightly/svs-shared-library-0.5.0rc2-reduced-gcc14.tar.gz" CACHE STRING "SVS URL")
else()
set(SVS_URL "https://github.com/intel/ScalableVectorSearch/releases/download/v0.3.2/svs-shared-library-reduced.tar.gz" CACHE STRING "SVS URL")
set(SVS_URL "https://github.com/intel/ScalableVectorSearch/releases/download/nightly/svs-shared-library-0.5.0rc2-reduced.tar.gz" CACHE STRING "SVS URL")
endif()
elseif(GLIBC_VERSION VERSION_GREATER_EQUAL "2.26")
set(SVS_URL "https://github.com/intel/ScalableVectorSearch/releases/download/v0.3.0/svs-shared-library-0.3.0-reduced-glibc2_26.tar.gz" CACHE STRING "SVS URL")
else()
message(STATUS "GLIBC >= 2.26 is required for GCC SVS binaries - disabling SVS_SHARED_LIB")
message(STATUS "GLIBC >= 2.28 is required for GCC SVS binaries - disabling SVS_SHARED_LIB")
set(SVS_SHARED_LIB OFF)
endif()
endif()
Expand Down Expand Up @@ -128,7 +126,7 @@ if(USE_SVS)
# so whether it is available depends on which SVS this build ended up with rather than on the
# platform. Detect it in the header instead of assuming: it is a method, not a file, so this
# greps the header the LVQ check would have tested for existence.
set(SVS_DYNAMIC_INDEX_HEADER "svs/index/vamana/dynamic_index.h")
set(SVS_DYNAMIC_INDEX_HEADER "svs/concurrent/dynamic_index.h")
set(SVS_HAS_REPLACE_EXTERNAL_ID 0)
if(EXISTS "${svs_SOURCE_DIR}/include/${SVS_DYNAMIC_INDEX_HEADER}")
file(READ "${svs_SOURCE_DIR}/include/${SVS_DYNAMIC_INDEX_HEADER}" SVS_DYNAMIC_INDEX_SRC)
Expand Down
2 changes: 1 addition & 1 deletion deps/ScalableVectorSearch
Submodule ScalableVectorSearch updated 71 files
+4 −1 bindings/c/CMakeLists.txt
+98 −10 bindings/c/docs/C_API_Design.md
+241 −0 bindings/c/include/svs/c/svs_c.h
+5 −0 bindings/c/src/algorithm.hpp
+142 −0 bindings/c/src/allocator.hpp
+133 −0 bindings/c/src/data_builder.cpp
+8 −0 bindings/c/src/data_builder.hpp
+61 −4 bindings/c/src/data_builder/leanvec.hpp
+66 −8 bindings/c/src/data_builder/lvq.hpp
+13 −2 bindings/c/src/data_builder/simple.hpp
+19 −2 bindings/c/src/data_builder/sq.hpp
+73 −5 bindings/c/src/dispatcher_dynamic_vamana.cpp
+11 −0 bindings/c/src/dispatcher_dynamic_vamana.hpp
+57 −11 bindings/c/src/dispatcher_vamana.cpp
+13 −2 bindings/c/src/dispatcher_vamana.hpp
+11 −0 bindings/c/src/error.hpp
+45 −10 bindings/c/src/index.hpp
+171 −3 bindings/c/src/index_builder.hpp
+61 −0 bindings/c/src/leanvec_training_data.cpp
+85 −0 bindings/c/src/leanvec_training_data.hpp
+21 −3 bindings/c/src/storage.hpp
+312 −25 bindings/c/src/svs_c.cpp
+5 −0 bindings/c/src/threadpool.hpp
+24 −0 bindings/c/src/types_support.hpp
+460 −1 bindings/c/tests/c_api_dynamic_index.cpp
+516 −0 bindings/c/tests/c_api_index.cpp
+100 −1 bindings/c/tests/c_api_test_utils.h
+11 −7 bindings/cpp/include/svs/runtime/vamana_index.h
+1 −0 bindings/cpp/src/vamana_index_impl.h
+288 −0 include/svs/concurrent/README.md
+433 −0 include/svs/concurrent/blocked_data.h
+54 −0 include/svs/concurrent/concurrent.h
+524 −0 include/svs/concurrent/consolidate.h
+2,115 −0 include/svs/concurrent/dynamic_index.h
+625 −0 include/svs/concurrent/dynamic_search_buffer.h
+54 −0 include/svs/concurrent/extensions/scalar.h
+740 −0 include/svs/concurrent/graph.h
+222 −0 include/svs/concurrent/graph_concepts.h
+201 −0 include/svs/concurrent/greedy_search.h
+358 −0 include/svs/concurrent/iterator.h
+979 −0 include/svs/concurrent/multi.h
+338 −0 include/svs/concurrent/prune.h
+142 −0 include/svs/concurrent/reverse_edges.h
+49 −0 include/svs/concurrent/spinlock.h
+495 −0 include/svs/concurrent/translation.h
+605 −0 include/svs/concurrent/vamana_build.h
+32 −26 include/svs/core/allocator.h
+5 −3 include/svs/core/data.h
+15 −14 include/svs/core/data/simple.h
+11 −5 include/svs/core/graph.h
+105 −10 include/svs/index/vamana/dynamic_index.h
+38 −3 include/svs/index/vamana/dynamic_search_buffer.h
+12 −10 include/svs/index/vamana/index.h
+45 −0 include/svs/index/vamana/search_buffer.h
+93 −0 include/svs/lib/concurrency/atomic_span.h
+141 −0 include/svs/lib/concurrency/seqlock.h
+371 −0 include/svs/lib/segmented_vector.h
+26 −4 include/svs/orchestrators/dynamic_vamana.h
+11 −7 include/svs/orchestrators/vamana.h
+78 −0 tests/CMakeLists.txt
+578 −0 tests/svs/concurrent/concurrency.cpp
+82 −0 tests/svs/concurrent/consolidate.cpp
+161 −0 tests/svs/concurrent/downstream.cpp
+562 −0 tests/svs/concurrent/dynamic_index.cpp
+727 −0 tests/svs/concurrent/dynamic_index_2.cpp
+129 −0 tests/svs/concurrent/graph.cpp
+421 −0 tests/svs/concurrent/iterator.cpp
+475 −0 tests/svs/concurrent/multi.cpp
+59 −0 tests/svs/concurrent/prune.cpp
+288 −0 tests/svs/concurrent/translation.cpp
+237 −0 tests/svs/lib/segmented_vector.cpp
214 changes: 43 additions & 171 deletions src/VecSim/algorithms/hnsw/hnsw_tiered.h

Large diffs are not rendered by default.

Loading
Loading