Skip to content

Commit 532065c

Browse files
pockeclaude
authored andcommitted
Preserve UTF-8 encoding when reallocating a frozen ResumableParser buffer
`JSON::ResumableParser#<<` raises `Encoding::CompatibilityError` when the first chunk is a frozen multibyte UTF-8 string and more data is fed after a partial parse. ## Reproduction ```ruby require "json" parser = JSON::ResumableParser.new parser << '{"message":"日本'.freeze # frozen + multibyte (UTF-8) parser.parse # => false (incomplete) parser << '語のつづき"}'.freeze # => Encoding::CompatibilityError: BINARY and UTF-8 ``` The error only surfaces when the first chunk is both frozen and multibyte. A frozen ASCII-only first chunk stays ASCII-compatible and a mutable first chunk keeps its encoding, so neither triggers it. This is common in streaming use cases where chunks arrive as frozen UTF-8 (e.g. gRPC/protobuf string fields), and is easily hit with multibyte (e.g. Japanese) payloads. ## Root cause In `cResumableParser_feed`, the first feed adopts a frozen input string directly as the buffer (`parser->buffer = str`), keeping its UTF-8 encoding. When a later feed needs to reallocate that frozen buffer, it allocates the new buffer with `rb_str_buf_new()`, which returns an ASCII-8BIT (BINARY) string and does not carry over the original UTF-8 encoding. The subsequent `rb_str_append(parser->buffer, str)` then appends a multibyte UTF-8 chunk to a BINARY buffer that already holds non-ASCII bytes, and the two encodings are incompatible. `convert_encoding` always normalizes the input to UTF-8, so the buffer is guaranteed to be UTF-8. Associating the freshly allocated buffer with UTF-8 restores that invariant. This mirrors the existing idiom used for the generator buffer, and also fixes the latent case where a frozen ASCII-only first chunk would silently leave the buffer as BINARY. ## Test Added a regression test feeding frozen multibyte chunks across a partial parse. The byte-by-byte `assert_resumed_parsing` helper cannot cover this: it feeds single ASCII-8BIT bytes via `byte.chr`, which `convert_encoding` turns into a mutable dup, so the buffer never becomes frozen and the reallocation path is never taken. The `.freeze` calls are kept explicit so the test still exercises the frozen path even if the file's `frozen_string_literal` magic comment is ever removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PpeQSEFkF1X1Uuzjx9BsYe
1 parent 7c8af4b commit 532065c

2 files changed

Lines changed: 11 additions & 0 deletions

File tree

ext/json/ext/parser/parser.c

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2388,6 +2388,7 @@ static VALUE cResumableParser_feed(VALUE self, VALUE str)
23882388

23892389
if (RB_OBJ_FROZEN_RAW(parser->buffer)) {
23902390
VALUE new_buffer = rb_obj_hide(rb_str_buf_new(remaining + RSTRING_LEN(str)));
2391+
rb_enc_associate_index(new_buffer, utf8_encindex);
23912392

23922393
char *old_ptr = RSTRING_PTR(parser->buffer);
23932394
memcpy(RSTRING_PTR(new_buffer), old_ptr + consumed, remaining);

test/json/resumable_parser_test.rb

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -205,6 +205,16 @@ def test_rest
205205
assert_equal '"unterminated string', @parser.rest
206206
end
207207

208+
def test_feed_frozen_multibyte_chunks
209+
@parser << '{"message":"日本'.freeze
210+
refute @parser.parse
211+
@parser << '語のつづき"}'.freeze
212+
assert @parser.parse
213+
value = @parser.value
214+
assert_equal({ "message" => "日本語のつづき" }, value)
215+
assert_equal Encoding::UTF_8, value["message"].encoding
216+
end
217+
208218
def test_eos
209219
assert_predicate @parser, :eos?
210220

0 commit comments

Comments
 (0)