initializationData not correct when there is null character in ass subtitle.

[peerless2012]

Version

Media3 1.8.0

More version details

Release v1.8.0

Devices that reproduce the issue

Pixel 6a

Devices that do not reproduce the issue

No

Reproducible in the demo app?

Yes

Reproduction steps

Add test mkv and play.

Expected result

initializationData has the correct data

Actual result

initializationData data only has part of data, and the data break due to the zero data before [Events] tag.

Media

See:

• ASS subtitles embedded in MKV files sometimes don't display
• ASS encoder inserts null character into middle of output
• Null Character in ASS subtitle Prevents Subtitles from Rendering

Bug Report

• You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.

[ethaldeman]

This issue was prompted by my issue on libass-andorid at peerless2012/libass-android#55 . Attached is a zip of an mkv that can be used to test this issues.

broken_ass_subtitles.zip

[icbaker]

Based on the description, I suspect the behaviour you're describing is caused by this fix/workaround where we encountered null-terminated strings created by handbrake (more details in the PR linked from the commit description): google/ExoPlayer@74bbd53

It looks like handbrake have fixed the logic that introduces these nulls: HandBrake/HandBrake#3258 (comment)

But of course the problematic media may still be in the ecosystem. So now we have some files that expect us to stop reading strings at NUL, and the file provided in this issue that expect us to read past the NUL byte. I'm not sure how we reconcile that.

(I will bookmark this issue to remind myself why we usually push back on being tolerant of media that doesn't seem to be quite following the relevant spec)

[ethaldeman]

I stumbled upon this issue while trying to test libass-andorid’s integration into the jellyfin android tv app. Libass-android was added to jellyfin-addind in the master branch only, but withheld from the debug and release apks. One of the major contributing reasons was multiple testers reported libass-android would not render subtitles for a significant portion of media, but vlc, mpv, ffmpeg didn’t have any issues rendering them for this media.

I believe I have tracked it down to this issue. My media collection has about ~200 pieces of media with ASS subtitles. ~35 of them won’t render with libass-android and I have confirmed it is caused by this issue. The ffmpeg issue https://fftrac-bg.ffmpeg.org/ticket/10203 where ffmpeg addresses this issue also suggests that the issue is wide spread, and ffmpeg accepted a patch to fix it. The post also suggests the null character may just be an unclear part of the 'spec'. The null character appears between the header portion and actual sub style portion of the file.

I fully understand working with ASS subtitles can be difficult, because it isn’t as much a spec but a general guideline, however I do believe this is an issue exoplayer needs to address. While a solution that address both this issue and google/ExoPlayer#8265 would be ideal, if it is only feasible to solve one, I think it should be this one.

[icbaker]

Taking a closer look at the issue reported here, which seems to be specific to the Format.initializationData value provided by MatroskaExtractor, I don't think the behaviour being flagged is related to the change made in google/ExoPlayer@74bbd53 (since that only affects subtitle sample data, not the data passed from matroska codec private to Format.initializationData.

The provided file has an SSA track with 521 bytes of codec private data:

$ mkvinfo -v ghi-2800.mkv
...
| + Track
|  + Track number: 3 (track ID for mkvmerge & mkvextract: 2)
|  + Track UID: 15795730791595501851
|  + "Lacing" flag: 0
|  + Language: und
|  + "Default track" flag: 0
|  + Codec ID: S_TEXT/ASS
|  + Track type: subtitles
|  + Maximum block additional ID: 0
|  + Codec's private data: size 521

I added some logging where this codec private data is passed as format.initializationData into the SubtitleParser.Factory:

media/libraries/extractor/src/main/java/androidx/media3/extractor/text/SubtitleTranscodingTrackOutput.java

Line 95 in bfe5930

? subtitleParserFactory.create(format)

And I see all 521 bytes (and the prefixed hard-coded line):

Creating subs parser for format=id=3, mimeType=text/x-ssa, container=video/x-matroska, language=und, initData.length=[90, 521], initData=[Format: Start, End, ReadOrder, Layer, Style, Name, MarginL, MarginR, MarginV, Effect, Text, [Script Info]
ScriptType: v4.00+                                                                                                                                                                                                                                 
Collisions: Normal                                                                                                                                                                                                                                 
PlayResX: 1920                                                                                                                                                                                                                                     
PlayResY: 1080                                                                                                                                                                                                                                     
Timer: 100.0                                                                                                                                                                                                                                       
WrapStyle: 0                                                                                                                                                                                                                                       
ScaledBorderAndShadow: yes                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                   
[V4+ Styles]                                                                                                                                                                                                                                       
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding      
Style: Default,sans-serif,71,&H00FFFFFF,&H00FFFFFF,&H000F0F0F,&H000F0F0F,0,0,0,0,100,100,0,0.00,1,2.37,1.97,2,20,20,20,0                                                                                                                           
��]                                                                                                                                                                                                                                                

It does include the null byte at the end (as that's present in the file, and included in the 521 bytes of the matroska declared length).

---

So please can you clarify what you expect here? Everything seems to be working as expected from my understanding of how MatroskaExtractor should be populating Format.initializationData.

[peerless2012]

Because the initializationData lost the event tag after the null character, the libass will not render, because of the lost event data.

I'm thinking if it is possible to read the whole data, and drop the null character.

[icbaker]

I don't see the event tag in the codec private data of the file when inspecting it with a hex editor. I wouldn't expect to see events in the initialization data (derived from the codec private data). The events would be in timed samples instead. From https://www.matroska.org/technical/subtitles.html (emphasis mine):

Now, how are they stored in Matroska?

• All text is converted to UTF-8
• All the headers are stored in CodecPrivate (Script Info and the Styles list)
• Start & End field are used to set TimeStamp and the BlockDuration element. the data stored is:
• Events are stored in the Block in this order: ReadOrder, Layer, Style, Name, MarginL, MarginR, MarginV, Effect, Text (Layer comes from ASS specs … it’s empty for SSA.) “ReadOrder field is needed for the decoder to be able to reorder the streamed samples as they were placed originally in the file.”

Can you clarify why you expect to see events in the initialization data?

[peerless2012]

I will update this issue next weekend, because we are in national day.

[peerless2012]

@icbaker
The initialization[1] should be 613 bytes, not 519 bytes.

and the 613 bytes contains the Event, I delete the null character and remux it to mkv.

[Script Info]
ScriptType: v4.00+
Collisions: Normal
PlayResX: 720
PlayResY: 540
Timer: 100.0
WrapStyle: 0
ScaledBorderAndShadow: yes

[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: Default,sans-serif,35,&H00FFFFFF,&H00FFFFFF,&H000F0F0F,&H000F0F0F,0,0,0,0,100,100,0,0.00,1,1.17,0.97,2,20,20,20,0

[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text

But the 519 bytes is the original ass in the test mkv.

[Script Info]
ScriptType: v4.00+
Collisions: Normal
PlayResX: 720
PlayResY: 540
Timer: 100.0
WrapStyle: 0
ScaledBorderAndShadow: yes

[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: Default,sans-serif,35,&H00FFFFFF,&H00FFFFFF,&H000F0F0F,&H000F0F0F,0,0,0,0,100,100,0,0.00,1,1.17,0.97,2,20,20,20,0
NUL