Pull Vector Set implementation into dev

[kevin-montrose]

Brings Vector Set implementation into dev, moving to Namespace and RecordType in store v2 instead of the hackery on top of store v1.

Meaningful differences:

• VectorElementKey
• Setting RecordType
• A prefix byte to indicate replication type, since shipping LogRecord isn't sufficient

Store V2 makes unaligned values a possibility, which violates some guarantees Vector Sets made to DiskANN. I paper over this by over-allocating some padding and adjusting things before calling into DiskANN. This does complicate VectorSessionFunctions and replication, as those padding bytes must be accounted for. Eventually we can make alignment a Tsavorite thing, and then all this disappears.

[Copilot]

Pull request overview

This PR brings the Vector Set implementation into the dev branch by wiring vector-set storage into Store v2 semantics (namespace + record type), and integrating vector operations with RESP command handling, replication, and cluster migration.

Changes:

• Adds Vector Set command support end-to-end (RESP parsing/routing, storage/session ops, API surface, ACL categories, and server options gating).
• Extends Tsavorite record metadata/initialization to carry a RecordType and enables additional status helpers/pinning utilities.
• Updates cluster migration/replication flows to support Vector Sets (typed migration payloads, CLUSTER RESERVE, replica background tasks, and slot-verification nuances).

Reviewed changes

Copilot reviewed 112 out of 117 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test/Garnet.test/TestUtils.cs	Adds test helpers/options for vector preview enablement and AOF memory sizing.
test/Garnet.test/Resp/ACL/RespCommandTests.cs	Adds ACL coverage for new vector commands and CLUSTER RESERVE.
test/Garnet.test/ReadOptimizedLockTests.cs	New tests for ReadOptimizedLock behavior and concurrency.
test/Garnet.test/DiskANN/DiskANNGridTests.cs	Adds (currently ignored) grid-based vector search tests.
test/Garnet.test/CountingEventSlimTests.cs	Adds tests for new CountingEventSlim.
test/Garnet.test.cluster/ClusterTestUtils.cs	Adds cancellation token support for migration cleanup waiting.
playground/Bitmap/BitOp.cs	Fixes label/region indentation consistency.
libs/storage/Tsavorite/cs/src/core/VarLen/RecordSizeInfo.cs	Makes CalculateSizes public to support external callers.
libs/storage/Tsavorite/cs/src/core/VarLen/RecordFieldInfo.cs	Adds RecordType to field metadata and includes it in ToString().
libs/storage/Tsavorite/cs/src/core/Utilities/Status.cs	Adds CreateNotFound() / CreateError() helpers.
libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/Implementation/ContinuePending.cs	Formatting/indentation fix.
libs/storage/Tsavorite/cs/src/core/Allocator/OverflowByteArray.cs	Adds Pin() helper to return a pinned GCHandle.
libs/storage/Tsavorite/cs/src/core/Allocator/ObjectSerialization/RecordDataHeader.cs	Drives record type from RecordSizeInfo.FieldInfo.RecordType; adjusts namespace-length behavior.
libs/storage/Tsavorite/cs/src/core/Allocator/LogRecord.cs	Updates header initialization call sites after RecordDataHeader.Initialize signature change.
libs/storage/Tsavorite/cs/src/core/Allocator/AllocatorScan.cs	Formatting/indentation fix.
libs/server/VectorOutput.cs	Introduces VectorOutput wrapper type for vector store outputs.
libs/server/Transaction/TxnKeyManager.cs	Slot verification updated to disable “wait for stable slot” for txn ownership checks.
libs/server/TaskManager/TaskType.cs	Adds replica-only VectorReplicationReplayTask placement.
libs/server/StoreWrapper.cs	Adds post-recovery vector set recovery and starts replica-specific tasks when applicable.
libs/server/Storage/Session/UnifiedStore/UnifiedStoreOps.cs	Extends DELETE to fall back to vector-set deletion on canceled delete.
libs/server/Storage/Session/StorageSession.cs	Adds vector session contexts and injects VectorManager into StorageSession.
libs/server/Storage/Session/MainStore/VectorStoreOps.cs	Adds main-store vector set operations (VADD/VREM/VSIM/etc).
libs/server/Storage/Session/MainStore/MainStoreOps.cs	Maps wrong-type reads to GarnetStatus.WRONGTYPE for GET.
libs/server/Storage/Session/MainStore/AdvancedOps.cs	Maps wrong-type reads to GarnetStatus.WRONGTYPE for advanced reads.
libs/server/Storage/Functions/UnifiedStore/DeleteMethods.cs	Cancels unified-store delete when vector index requires special handling.
libs/server/Storage/Functions/MainStore/VarLenInputMethods.cs	Sets record type/size for vector index records; adjusts modified field info for vector ops.
libs/server/Storage/Functions/MainStore/RMWMethods.cs	Adds VADD/VREM special handling for (synthetic) replication/index recreation/deleteability.
libs/server/Storage/Functions/MainStore/ReadMethods.cs	Adds vector-set wrong-type detection and GET “hack” using arg1 < 0 marker.
libs/server/Storage/Functions/MainStore/PrivateMethods.cs	Ensures certain commands read raw value; suppresses AOF logging for non-synthetic VADD.
libs/server/Storage/Functions/FunctionsState.cs	Stores VectorManager in functions state for session-function access.
libs/server/Servers/GarnetServerOptions.cs	Adds EnableVectorSetPreview feature gate.
libs/server/Resp/Vector/VectorManager.Migration.cs	Adds vector-set migration handling (element/index transfer + replication synthesis).
libs/server/Resp/Vector/VectorManager.Index.cs	Defines vector index value layout and helpers to read/update it for migration/recreate.
libs/server/Resp/Vector/VectorManager.Filter.cs	Adds zero-alloc post-filtering pipeline for VSIM filter evaluation.
libs/server/Resp/Vector/VectorManager.Callbacks.cs	Adds DiskANN callbacks and vector batch reads bridging unmanaged callbacks to Tsavorite.
libs/server/Resp/Vector/ExprRunner.cs	Implements expression VM for filtering (stack-based, UTF-8 span comparisons).
libs/server/Resp/RespServerSessionSlotVerify.cs	Forces “wait for stable slot” for vector write-like commands.
libs/server/Resp/RespServerSession.cs	Routes vector commands and injects vector contexts into cluster session; fixes DB switching to use per-DB VectorManager.
libs/server/Resp/RespCommandInfoFlags.cs	Adds Module command flag and Vector ACL category.
libs/server/Resp/Parser/SessionParseState.cs	Adds EnsureCapacity and float parsing helpers.
libs/server/Resp/Parser/RespCommand.cs	Adds vector commands, CLUSTER RESERVE, and vector-set legality helper; extends fast parsing.
libs/server/Resp/Parser/ParseUtils.cs	Adds float parsing helpers and infinity support.
libs/server/Resp/MGetReadArgBatch.cs	Marks internal GETs via StringInput(RespCommand.GET, arg1: -1) to bypass vector-type checks.
libs/server/Resp/LocalServerSession.cs	Exposes VectorBasicContext from a local session.
libs/server/Resp/CmdStrings.cs	Adds RESERVE subcommand string.
libs/server/Resp/BasicCommands.cs	Marks network GET as RESP GET (arg1: -1) and writes WRONGTYPE errors explicitly.
libs/server/Resp/AdminCommands.cs	Passes VectorManager into cluster command processing.
libs/server/Lua/LuaTimeoutManager.cs	Formatting/indentation fix.
libs/server/InputHeader.cs	Introduces VectorInput store input type for vector element operations.
libs/server/GarnetDatabase.cs	Adds per-DB VectorManager and disposes it during DB disposal.
libs/server/Garnet.server.csproj	Adds diskann-garnet package dependency.
libs/server/Databases/SingleDatabaseManager.cs	Initializes/resumes vector manager during recovery and post-recovery steps; passes it into FunctionsState.
libs/server/Databases/MultiDatabaseManager.cs	Same as single-DB manager but per active DB.
libs/server/Databases/IDatabaseManager.cs	Adds RecoverVectorSets() contract.
libs/server/Databases/DatabaseManagerBase.cs	Passes VectorManager when creating storage sessions; adds RecoverVectorSets() abstract method.
libs/server/Cluster/IClusterSession.cs	Extends cluster APIs to accept vector manager and stable-slot waiting.
libs/server/Cluster/IClusterProvider.cs	Extends cluster session creation to receive string/vector contexts.
libs/server/Cluster/ClusterSlotVerificationInput.cs	Adds waitForStableSlot flag used during migration.
libs/server/ArgSlice/ArgSliceVector.cs	Extends arg-slice buffering to support (namespace,key) tuples for vector element migration.
libs/server/API/IGarnetApi.cs	Adds vector set APIs (add/remove + similarity/metadata operations).
libs/server/API/GarnetWatchApi.cs	Adds WATCH-wrapped vector set API methods.
libs/server/API/GarnetStatus.cs	Adds BADSTATE status code.
libs/server/API/GarnetApi.cs	Implements vector set API methods by delegating to StorageSession.
libs/server/AOF/AofProcessor.cs	Adds vector-aware replay handling and ensures vector ops complete when needed.
libs/server/ACL/ACLParser.cs	Adds “vector” ACL category parsing.
libs/resources/RespCommandsInfo.json	Registers vector commands and `CLUSTER
libs/resources/RespCommandsDocs.json	Adds docs entries for vector commands.
libs/host/GarnetServer.cs	Creates per-DB VectorManager during DB creation.
libs/host/Garnet.host.csproj	Adds diskann-garnet package dependency.
libs/GlobalUsings.cs	Adds global using aliases for string/vector contexts (basic + transactional).
libs/common/VectorElementKey.cs	Introduces VectorElementKey key type (always namespaced).
libs/common/RespReadUtils.cs	Adds float infinity parsing.
libs/common/GarnetKeyComparer.cs	Adds equality/hash special-casing for VectorElementKey.
libs/common/FixedSpanByteKey.cs	Minor formatting change.
libs/common/ExceptionInjectionType.cs	Adds vector set delete interruption injection points.
libs/common/CountingEventSlim.cs	Adds CountingEventSlim synchronization primitive.
libs/cluster/Session/SlotVerification/RespClusterSlotVerify.cs	Propagates stable-slot waiting option through multi-key slot verification.
libs/cluster/Session/SlotVerification/RespClusterIterativeSlotVerify.cs	Adds stable-slot waiting option to iterative verification.
libs/cluster/Session/SlotVerification/ClusterSlotVerify.cs	Implements stable-slot waiting for vector writes during migration/importing.
libs/cluster/Session/RespClusterReplicationCommands.cs	Adds CLUSTER RESERVE (vector set context reservations).
libs/cluster/Session/RespClusterMigrateCommands.cs	Extends MIGRATE to carry vector-set migration payloads and apply them appropriately.
libs/cluster/Session/ReplicaOfCommand.cs	Starts/stops replica-only tasks when role changes.
libs/cluster/Session/ClusterSession.cs	Wires in contexts and passes vector manager to cluster command processing.
libs/cluster/Session/ClusterCommands.cs	Routes CLUSTER RESERVE.
libs/cluster/Server/Replication/PrimaryOps/DisklessReplication/ReplicationSnapshotIterator.cs	Adds record-span typing for migration/replication streaming.
libs/cluster/Server/Replication/PrimaryOps/DisklessReplication/ReplicaSyncSession.cs	Adds record-span typing to network writes.
libs/cluster/Server/Migration/Sketch.cs	Adds namespace-aware hashing/storage for vector element migration keys.
libs/cluster/Server/Migration/MigrationDriver.cs	Reserves destination vector contexts before migrating when vector namespaces are present.
libs/cluster/Server/Migration/MigrateSessionSlots.cs	Migrates vector namespaces + index keys; deletes vector sets after index transfer.
libs/cluster/Server/Migration/MigrateSessionKeys.cs	Extends key-based migration to handle vector namespaces + index transfer.
libs/cluster/Server/Migration/MigrateSessionCommonUtils.cs	Adds typed record-span writing and vector element record handling.
libs/cluster/Server/Migration/MigrateSession.cs	Adds namespace tracking/map for vector migration.
libs/cluster/Server/Migration/MigrateScanFunctions.cs	Detects namespaced vector elements and vector index records during scan.
libs/cluster/Server/Migration/MigrateOperation.cs	Tracks vector sets encountered and transmits namespaced vector element data.
libs/cluster/Server/Failover/ReplicaFailoverSession.cs	Stops replica-only tasks and starts primary tasks during replica failover.
libs/cluster/Server/ClusterProvider.cs	Passes contexts into cluster session creation.
libs/cluster/Server/ClusterManagerWorkerState.cs	Starts/stops replica/primary-only tasks when role changes.
libs/cluster/Server/ClusterManagerSlotState.cs	Removes unnecessary unsafe modifier.
libs/client/ClientSession/GarnetClientSessionMigrationExtensions.cs	Extends MIGRATE header to include vector-set mode flag.
libs/client/ClientSession/GarnetClientSessionIncremental.cs	Adds MigrationRecordSpanType and prepends payload type byte when writing record spans.
Directory.Packages.props	Adds centralized diskann-garnet package version.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[badrishc]

nit: shoulnd't -> shouldn't