Skip to content

Commit d070ab3

Browse files
committed
Updated the documents for KVcache with aoti support
1 parent 9d6a300 commit d070ab3

8 files changed

Lines changed: 517 additions & 772 deletions

File tree

corelib/recsys_kvcache_manager/README.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,11 @@ Furthermore, the `NativeHostKVCacheManager` backend supports layerwise KV data o
5050
`HostKVStorageManagerBase` is provided as an interface for other LLM-compatible KV-cache systems for host memory, storage, and remote data pools.
5151
This can be extended to integrate other KV-cache systems. Currently, we provide integration with [`FlexKV`](https://github.com/taco-project/FlexKV/tree/main) as the lower-tier KV storage backend.
5252

53+
- **[NEW] Compatible with Torch Export and AOTI**:
54+
Recsys KVCache Manager now includes an export-compatible backend for KV-cache lookup, allocation, onboarding, offloading, and eviction when using the paged GPU KV cache together with the FlexKV host cache backend.
55+
The export path keeps the Python API unchanged, but routes KV-cache operations through tensor-only torch custom ops while keeping stateful cache policy and runtime management in C++, which makes the flow compatible with `torch.export` and AOTInductor/AOTI packaging. The current implementation is dependent on a [customized FlexKV version](https://github.com/geoffreyQiu/FlexKV/tree/cpp_client).
56+
57+
5358

5459
## Installation
5560

corelib/recsys_kvcache_manager/EXPORT_KVCACHE_DESIGN.md renamed to corelib/recsys_kvcache_manager/doc/EXPORT_KVCACHE_DESIGN.md

File renamed without changes.

0 commit comments

Comments
 (0)