Commit 532065c
Preserve UTF-8 encoding when reallocating a frozen ResumableParser buffer
`JSON::ResumableParser#<<` raises `Encoding::CompatibilityError` when the
first chunk is a frozen multibyte UTF-8 string and more data is fed after a
partial parse.
## Reproduction
```ruby
require "json"
parser = JSON::ResumableParser.new
parser << '{"message":"日本'.freeze # frozen + multibyte (UTF-8)
parser.parse # => false (incomplete)
parser << '語のつづき"}'.freeze # => Encoding::CompatibilityError: BINARY and UTF-8
```
The error only surfaces when the first chunk is both frozen and multibyte.
A frozen ASCII-only first chunk stays ASCII-compatible and a mutable first
chunk keeps its encoding, so neither triggers it. This is common in
streaming use cases where chunks arrive as frozen UTF-8 (e.g. gRPC/protobuf
string fields), and is easily hit with multibyte (e.g. Japanese) payloads.
## Root cause
In `cResumableParser_feed`, the first feed adopts a frozen input string
directly as the buffer (`parser->buffer = str`), keeping its UTF-8 encoding.
When a later feed needs to reallocate that frozen buffer, it allocates the
new buffer with `rb_str_buf_new()`, which returns an ASCII-8BIT (BINARY)
string and does not carry over the original UTF-8 encoding. The subsequent
`rb_str_append(parser->buffer, str)` then appends a multibyte UTF-8 chunk to
a BINARY buffer that already holds non-ASCII bytes, and the two
encodings are incompatible.
`convert_encoding` always normalizes the input to UTF-8, so the buffer is
guaranteed to be UTF-8. Associating the freshly allocated buffer with UTF-8
restores that invariant. This mirrors the existing idiom used for the
generator buffer, and also fixes the latent case where a frozen ASCII-only
first chunk would silently leave the buffer as BINARY.
## Test
Added a regression test feeding frozen multibyte chunks across a partial
parse. The byte-by-byte `assert_resumed_parsing` helper cannot cover this:
it feeds single ASCII-8BIT bytes via `byte.chr`, which `convert_encoding`
turns into a mutable dup, so the buffer never becomes frozen and the
reallocation path is never taken. The `.freeze` calls are kept explicit so
the test still exercises the frozen path even if the file's
`frozen_string_literal` magic comment is ever removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PpeQSEFkF1X1Uuzjx9BsYe1 parent 7c8af4b commit 532065c
2 files changed
Lines changed: 11 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2388 | 2388 | | |
2389 | 2389 | | |
2390 | 2390 | | |
| 2391 | + | |
2391 | 2392 | | |
2392 | 2393 | | |
2393 | 2394 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
205 | 205 | | |
206 | 206 | | |
207 | 207 | | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
208 | 218 | | |
209 | 219 | | |
210 | 220 | | |
| |||
0 commit comments