Skip to content

Ignore quotes when detecting the row separator - #365

Open
gaoflow wants to merge 1 commit into
ruby:mainfrom
gaoflow:quote-aware-row-sep-detection
Open

Ignore quotes when detecting the row separator#365
gaoflow wants to merge 1 commit into
ruby:mainfrom
gaoflow:quote-aware-row-sep-detection

Conversation

@gaoflow

@gaoflow gaoflow commented Jul 28, 2026

Copy link
Copy Markdown

:auto row-separator detection scans the sample for the first CR/LF without tracking quote state, so a CR inside a quoted field is mistaken for the row separator. CSV therefore cannot parse its own generate output whenever a field contains \r:

CSV.parse(CSV.generate {|c| c << ["a\rb", "x"]})
# => [["a\rb", "x\n"]]   # wrong: the \n was swallowed into the last field
CSV.parse(CSV.generate {|c| c << ["a\r\nb", "x"]})
# => CSV::MalformedCSVError: Unquoted fields do not allow new line
CSV.parse(CSV.generate {|c| c << ["\r"]})
# => CSV::MalformedCSVError: Any value after quoted field isn't allowed

All three parse correctly with an explicit row_sep: "\n", which isolates the fault to :auto detection. detect_row_separator now skips CR/LF that fall inside a quoted field. An unquoted CR is still honored as a separator, so old Mac line endings keep working, and explicit row_sep paths are untouched.

Added round-trip and detection tests in test/csv/parse/test_row_separator.rb; the full suite stays green (527 tests, 4050 assertions). This is distinct from #107, which is about an explicit non-standard row_sep.

`:auto` row separator detection scanned the sample for the first CR/LF
without tracking quote state, so a CR inside a quoted field was mistaken
for the row separator. As a result CSV could not parse its own output:
`CSV.parse(CSV.generate)` corrupted or crashed on any field containing
a carriage return. For example `["a\rb", "x"]` round-tripped to
`["a\rb", "x\n"]`, and fields with `\r\n` or a lone `\r` raised
CSV::MalformedCSVError.

Make detection skip CR/LF that fall inside a quoted field. Unquoted CR
is still honored as a separator, so old Mac line endings keep working.
Comment thread lib/csv/parser.rb

# Index of the first +target+ that is outside a quoted field, so a CR or
# LF inside quotes is not mistaken for the row separator.
def unquoted_index(sample, target)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Could you rename unquoted_index to find_unquouted_string_index or something? I feel that no verb method name is strange for this because the receiver is Parser not String.
  • Could you rename sample to text or string because sample is strange in this context?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants