Skip to content

Commit d8b8eff

Browse files
committed
fix device-node cache aliasing null tag with literal string "null"
TsFileIOReader::get_cached_device_node keyed device_node_cache_ by IDeviceID::get_device_name(), which renders a null tag segment as the literal text "null". A device with a real null tag (e.g. tags (null, b, c)) and a device whose tag value is the string "null" (("null", b, c)) therefore produce the identical name "a.null.b.c" and collide in the cache: whichever device is queried first populates the entry, and every later query for the other device on the same reused reader gets the first device's cached MetaIndexNode and silently reads its chunks. This surfaced through the Python dataset API, where one long-lived TsFileReader answers many per-device queries: the pytest test_dataset_null_tag_positions_and_string_null_are_distinct read (null,b,c)'s data for the ("null",b,c) device. The device metadata binary search (DeviceIDComparable, segment-based) was always correct; only the string cache key was lossy. Key the cache by a collision-free, length-prefixed encoding of the segment vector that flags null segments explicitly, so a null tag can never alias the string "null". Add a device_id unit test pinning the invariant (names collide, segment equality distinguishes them). Verified: C++ suite 707/707 (Release+ASan+UBSan), Python suite 150/150.
1 parent 722d639 commit d8b8eff

3 files changed

Lines changed: 57 additions & 1 deletion

File tree

cpp/src/file/tsfile_io_reader.cc

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -384,10 +384,29 @@ int TsFileIOReader::load_tsfile_meta() {
384384
return ret;
385385
}
386386

387+
std::string TsFileIOReader::device_node_cache_key(
388+
const std::shared_ptr<IDeviceID>& device_id) {
389+
// Length-prefixed, null-flagged encoding: for each segment emit either
390+
// "N;" (null) or "<len>:<bytes>;". Distinct segment sequences always map
391+
// to distinct keys, so a real null tag never aliases the literal "null".
392+
std::string key;
393+
for (const std::string* seg : device_id->get_segments()) {
394+
if (seg == nullptr) {
395+
key += "N;";
396+
} else {
397+
key += std::to_string(seg->size());
398+
key += ':';
399+
key += *seg;
400+
key += ';';
401+
}
402+
}
403+
return key;
404+
}
405+
387406
int TsFileIOReader::get_cached_device_node(std::shared_ptr<IDeviceID> device_id,
388407
common::PageArena& pa,
389408
CachedDeviceNode& out) {
390-
std::string dev_name = device_id->get_device_name();
409+
std::string dev_name = device_node_cache_key(device_id);
391410

392411
{
393412
std::lock_guard<std::mutex> lk(device_node_cache_mu_);

cpp/src/file/tsfile_io_reader.h

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,6 +195,15 @@ class TsFileIOReader {
195195
common::PageArena& pa, CachedDeviceNode& out);
196196

197197
private:
198+
// Build a collision-free key for device_node_cache_. get_device_name()
199+
// renders a null tag segment as the literal "null", so a device with a
200+
// real null tag and one whose tag value is the string "null" produce the
201+
// same name and would alias in the cache — the second device would read
202+
// the first device's chunks. Encode each segment length-prefixed and
203+
// flag null segments explicitly so the two can never collide.
204+
static std::string device_node_cache_key(
205+
const std::shared_ptr<IDeviceID>& device_id);
206+
198207
ReadFile* read_file_;
199208
common::PageArena tsfile_meta_page_arena_;
200209
TsFileMeta tsfile_meta_;

cpp/test/common/device_id_test.cc

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,4 +71,32 @@ TEST(DeviceIdTest, TabletDeviceId) {
7171
ASSERT_EQ("test_device0.null.t2.t3",
7272
tablet.get_device_id(2)->get_device_name());
7373
}
74+
75+
// Regression: a device whose first tag is a real null and a device whose first
76+
// tag is the literal string "null" render to the SAME get_device_name()
77+
// ("t.null.b"), so anything that keys a per-device map/cache by the device name
78+
// aliases the two — the second device silently reads the first device's chunks.
79+
// The device-node cache in TsFileIOReader hit exactly this, conflating the two
80+
// devices' data on a reused reader. The reliable discriminator is the segment
81+
// vector (operator==), which keeps nullptr distinct from the string "null".
82+
TEST(DeviceIdTest, NullTagVsLiteralNullAreDistinct) {
83+
// Real null first tag: segment pointer is nullptr.
84+
std::vector<std::string*> null_first_segs{new std::string("t"), nullptr,
85+
new std::string("b")};
86+
StringArrayDeviceID null_first(null_first_segs);
87+
for (auto* s : null_first_segs) delete s;
88+
89+
// Literal string "null" as the first tag value.
90+
StringArrayDeviceID literal_null(
91+
std::vector<std::string>({"t", "null", "b"}));
92+
93+
// The names collide — this is the trap the cache used to fall into.
94+
ASSERT_EQ(null_first.get_device_name(), literal_null.get_device_name());
95+
ASSERT_EQ("t.null.b", null_first.get_device_name());
96+
97+
// But the devices are genuinely different, and the segment-based equality
98+
// used by DeviceIDComparable / the cache key must reflect that.
99+
ASSERT_FALSE(null_first == literal_null);
100+
ASSERT_TRUE(null_first != literal_null);
101+
}
74102
} // namespace storage

0 commit comments

Comments
 (0)