Support SSA/ASS styling

Add support for SSA/ASS styling. Currently there is only "minimal" support for ASS subtitles since styling was removed from the initial PR. (See #2281)

It would be great to have full support of the spec. Many subtitles use these styling features to work directly with the video (e.g. to substitute text in a video). SSA/ASS styling is also very common in certain genres of media like Cartoons, Anime or Karaoke.

@icbaker

@icbaker Tentatively marking as an enhancement, but please dupe or close as needed. Thanks!

Thanks for the request.

Unfortunately we don't currently have any capacity to improve ExoPlayer's SSA/ASS styling support. However we would welcome high quality PRs in this area.

I'm also happy to discuss various approaches with PR authors if a new feature doesn't fit easily into the existing code structure (either before the PR, or in the review, whichever is easier).

Could we determine a basic checklist of what isn't supported, to determine what features of the spec can be focused on in order of size and/or ease of implementation?

(I'm not familiar with what exactly even is supported currently).

Could we determine a basic checklist of what isn't supported, to determine what features of the spec can be focused on in order of size and/or ease of implementation?

That sounds like a good idea - the best way to do that is probably to compare the SsaDecoder to a copy of the spec. I've found the SSA/ASS spec to be fairly loose, and I'm not sure there's a canonical definition, but I've generally referred to this when working with these files in the past: http://moodub.free.fr/video/ass-specs.doc (linked from Matroska's page on SSA).

In case it's helpful, we would probably prioritise things roughly in this order:

Features that look like they're implemented but don't work properly (basically bugs).
Features that are similar to existing ones, and so can be implemented with small incremental changes.
Features that don't fit into the existing code architecture - so implementing them requires significant refactoring.

I don't know if SSA has anything in the final category - but if it looks like it does then please come talk to us before spending ages refactoring SsaDecoder, as we might have some ideas about the best way to approach the problem.

I'm not a developer, so I can't exactly contribute a lot, but proper support of ass/ssa in Exoplayer is something I'd very much would like to have, as Plex for Android currently makes me sad...
I don't know if this makes sense in the context of Exoplayer, but it seems to me that the closest thing we have to a "standard" for ass/ssa is with libass, as a lot of notable players have implemented it and it is actively developed. I think at least being compliant with how libass renders subtitles would be a good idea.

I don't know if this makes sense in the context of Exoplayer, but it seems to me that the closest thing we have to a "standard" for ass/ssa is with libass, as a lot of notable players have implemented it and it is actively developed. I think at least being compliant with how libass renders subtitles would be a good idea.

Thanks for the pointer to libass, I took a quick look. I think you're possibly suggesting two alternative things:

ExoPlayer could depend on libass and use it's logic for all the SSA rendering.
People working on improving ExoPlayer's existing SSA rendering logic could use libass as a reference implementation to see how different pieces of content should be rendered, as an augmentation to the "spec".

In case anyone is considering trying the first of these, I'd like to pre-emptively warn against it. libass looks like it's written in C, and we're very reluctant to pull in native/JNI dependencies unless a pure Java implementation of the same behaviour is deemed ~impossible. Given SSA/ASS is a relatively simple plaintext-based format, it seems completely feasible to parse it fully using pure Java - so it's extremely unlikely we'd accept a PR that introduced a native dependency to achieve this. There's more discussion about (not) using native deps for parsing subtitles in #7746 and #7749.

The second suggestion (use it as a reference implementation) makes perfect sense - and afaict it's used by VLC, so it should be enough to just play any content with VLC and compare to ExoPlayer's behaviour.

I leave that up to actual developers to decide. I just thought I'd point out that libass exists and is probably the best reference we have, and that rendering subs differently from libass could potentially make people like me wonder why the subs behave differently in ExoPlayer. The second alternative is probably the better option for ExoPlayer. :)

@icbaker

hey @icbaker , I'll check what the current SsaDecoder supports and what is missing in regards of styling, based on the specs you linked above, then I could probably start and implement those step by step (with my limited time).

@jakobkukla did you have anything specific in mind? I mean is there a specific styling option which you would like to see first?

Thanks for taking a look! Breaking down the work into small PRs definitely sounds good.

I'd love to get this implemented as it would make this player the only one with DV and proper .ass support in Android besides Kodi which is way too heavy for anything other than an Nvidia Shield.

Looking forward to this!

L

@icbaker I looked into what SSA/ASS supports and what is implemented and what not. So what I think about it. Currently there are two "subtitle" renderers. CanvasSubtitleOutput (the default) and WebViewSubtitleOutput (which I'm not sure how it's launched, I guess it's done by the app which uses ExoPlayer). Anyway - to play with it I forced the WebViewSubtitleOutput, just because doing some styling is easier for me personally with CSS. Many features ASS supports are not so easily done with CSS as well. I even played with doing stuff like BorderStyle, Outline, Shadow with SVGs, as CSS doesn't support BorderStyle-Outline the way ASS requires it. And I'm not even talking about Effects -- which are basically animations. I then went further down wanting to check how libass works and came across their project, which uses webassembly. Btw according to libass github page, Crunchyroll is using this project. I even entertained myself with using said webassembly with WebViewSubtitleOutput, but it's a no go. As it stands - ExoPlayer looks at subtitles as a series of cues being rendered on a WebView and/or Canvas. Libass creates just another "video" stream based on description of text file. This makes sense. Check this example page. It showcases some capabilities of SSA/ASS which I'm not sure are at all possible with cues. So what are next steps?

One can hope that someone ports libass to Java -- which, probably will never happen :)

Quoting you from above. I now think that parsing is indeed easy, but rendering is next to impossible (with cues).

libass looks like it's written in C, and we're very reluctant to pull in native/JNI dependencies unless a pure Java implementation of the same behaviour is deemed ~impossible. Given SSA/ASS is a relatively simple plaintext-based format, it seems completely feasible to parse it fully using pure Java

Keep it as it is -- as in, not ever adding the full list of features.

wdyt?

Good news, the libass for exoplayer is on the way, please check our progress in libass-android

Is there any progress on it? Many other multimedia apps are based on ExoPlayer and thus have limited support on ass/ssa subtitles.

You can find the libass for exoplayer at libass-android

As it stands - ExoPlayer looks at subtitles as a series of cues being rendered on a WebView and/or Canvas. Libass creates just another "video" stream based on description of text file.

This is why I suggested it might fit better to use libass to drive a custom effect (androidx/media#2053 (comment)) - my intuition is that plumbing the subtitle data into a custom effect will be easier than trying to pass video-like data through ExoPlayer's Cue machinery.

As it stands - ExoPlayer looks at subtitles as a series of cues being rendered on a WebView and/or Canvas. Libass creates just another "video" stream based on description of text file.

This is why I suggested it might fit better to use libass to drive a custom effect (androidx/media#2053 (comment)) - my intuition is that plumbing the subtitle data into a custom effect will be easier than trying to pass video-like data through ExoPlayer's Cue machinery.

Libass produce bitmap as output, witch is compat for Cue with bitmap. I already finish this with ARGB_888 bitmap.
But when I want to change bitmap to ALPHA_8, the Cue encode is error, and SubtitlePainter also need Blend bitmap.

Why do I stick with this approach?

Because this is a most simple way to render ass with libass, I don't need to worry about subtitles syncing with the video，and compat with current sutitle render process，with a few change in cue and SubtitlePainter.

If we use Effect (If it can work), there is more work I need todo, and to hack exo extractor and prase process.

@icbaker

@icbaker I'm currently trying to add LibassSubtitleRenderer, which would use libass, and eventually open a PR.
As you already mentioned, the Cue machinery doesn't "fit" how libass works.

If I am not mistaken, ASS/SSA subtitles are only available for the Matroska container, so all the decisions I made were based on this observation.

Libass Method	Media3 Function
`ass_render_frame`	`ExoPlayer.setVideoEffects()` + `BitmapOverlay`. I would return the bitmap generated by libass in `BitmapOverlay.getBitmap()`. Also, in the bitmap alpha blending, I would call `ExoPlayer.getVideoFormat().colorInfo` to do the "VSFilter mangled colors" explained here
`ass_set_frame_size`	Use the value obtained from `Player.getSurfaceSize()` + `Player.Listener.onSurfaceSizeChanged()`.
`ass_set_storage_size`	Use the value obtained from `Player.getVideoSize()` + `Player.Listener.onVideoSizeChanged()`.
`ass_process_codec_private`	`SsaParser` constructor, but `SsaParser` implements `SubtitleParser`, which requires me to use `Cues`, which I cannot do. So I'm not sure what I should do.
`ass_process_chunk`	`TrackOutput.sampleMetadata()` seems to receive the Matroska block, but `media3` seems to modify the block data and add the duration. It should be fine, but it is something I need to verify.
`ass_add_font` (Libass absolutely needs the font muxed into the MKV)	It is fairly simple to extract the font from the MKV (I did it here), but I don't see how I can pass the fonts to libass.

I saw your comment here:

This would involve a custom ExtractorsFactory and a customized ForwardingExtractorOutput that lets you intercept the subtitle data.

By default, Media3 uses DefaultExtractorsFactory, so I assume I would need to modify this class.

How could I add a special case to use libass in this class?

Also, I don't quite understand what I should edit, so this line returns my custom TrackOutput. Could you give me some detail?

and eventually open a PR.

Just to set expectations, based on the sketched design in your comment, I'm afraid we are unlikely to merge such a PR.

The design relies on media3's lib-effect, and libass native code, neither of which are currently depended on by lib-exoplayer, so either we'd need to add these deps (which then affects every user of the player library), or this would need to be in a new lib-ssa-parser module or similar, which comes with significant maintenance burden that I'm afraid probably isn't justified.

The design being so 'different' to all other subtitle parsers (by necessity) also increases the maintenance burden.

So I'm afraid a design like this will have to exist as an extension maintained outside the main media3 repository.

Support extract fonts in mkv and pass them to SubtitleParser maybe available?

In our project, we use AssMatroskaExtractor
witch extend MatroskaExtractor to extract the fonts attachments.

So I'm afraid a design like this will have to exist as an extension maintained outside the main media3 repository.

I'm not sure if this means it wouldn't be officially integrated into the player, but if that's the case even an unofficial fork would be great. I feel this is the only thing missing to have a player that "just works" not only on Android but on almost every other OS, since most have very big compromises, are too heavy or too hard to set up.

So I'm afraid a design like this will have to exist as an extension maintained outside the main media3 repository.

I'm not sure if this means it wouldn't be officially integrated into the player, but if that's the case even an unofficial fork would be great. I feel this is the only thing missing to have a player that "just works" not only on Android but on almost every other OS, since most have very big compromises, are too heavy or too hard to set up.

There is already a third plugin to bring libass in media3, you just need to add only one method to your players construct.

Here's an example:

player = ExoPlayer.Builder(this)
            .buildWithAssSupport(
                this,
                AssRenderType.OPEN_GL
            )

See buildWithAssSupport

and eventually open a PR.

Just to set expectations, based on the sketched design in your comment, I'm afraid we are unlikely to merge such a PR.

The design relies on media3's lib-effect, and libass native code, neither of which are currently depended on by lib-exoplayer, so either we'd need to add these deps (which then affects every user of the player library), or this would need to be in a new lib-ssa-parser module or similar, which comes with significant maintenance burden that I'm afraid probably isn't justified.

The design being so 'different' to all other subtitle parsers (by necessity) also increases the maintenance burden.

So I'm afraid a design like this will have to exist as an extension maintained outside the main media3 repository.

Honest question from someone very grateful exoplayer exists to begin with <3

Why is it unreasonable to include those dependencies?

If the goal is to support more subtitles which is a goal of accessibility I don't see how that's something that shouldn't be universal.

And before anyone says it... Text-only subs that don't rely on styling are not always an equal but not as pretty path. Quite frequently subs need to be placed in various different places and styling is often done to differentiate between characters, underline the mood/atmosphere /etc...

Text-only subs that don't rely on styling are not always an equal but not as pretty path. Quite frequently subs need to be placed in various different places and styling is often done to differentiate between characters, underline the mood/atmosphere /etc...

ExoPlayer already supports other formats than SSA/ASS that allow for positioning and styling. The argument for SSA/ASS is, as I understand it, to do with specific styling options that are only available in that format and that are particularly popular with some genres of content.

Text-only subs that don't rely on styling are not always an equal but not as pretty path. Quite frequently subs need to be placed in various different places and styling is often done to differentiate between characters, underline the mood/atmosphere /etc...

ExoPlayer already supports other formats than SSA/ASS that allow for positioning and styling. The argument for SSA/ASS is, as I understand it, to do with specific styling options that are only available in that format and that are particularly popular with some genres of content.

Correct, however when all that you have is SSA/ASS for a given piece of content (at least properly detailed subs, a lot of times rather mediocre compromise subs exist in text-only) then wishing for the other options getting used will get you nowhere.

How many genres are affected doesn't change that this does make some content less accessible.

I hope there are ways to get exoplayer to have proper as subtitles soon especially on Android tv. I use Artemio and exoplayer can play far more videos than libvlc, however the ass subtitles are poor and I can't read them as they are white with no borders. And their "ass support" feature does not work. Are there any exo based external players that have these features?

Navigation Menu

Repository navigation

Support SSA/ASS styling #8435

46 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support SSA/ASS styling #8435

Description

Activity

andrewlewis commented on Jan 7, 2021

icbaker commented on Jan 7, 2021

13steinj commented on Jan 8, 2021

icbaker commented on Jan 8, 2021

cosmo321 commented on Jan 13, 2021

icbaker commented on Jan 13, 2021

cosmo321 commented on Jan 13, 2021

szaboa commented on Jan 14, 2021

icbaker commented on Jan 15, 2021

46 remaining items

e1ker commented on May 13, 2024

zhuhang-jasper commented on Dec 21, 2024

peerless2012 commented on Jan 21, 2025

peerless2012 commented on Jan 21, 2025

icbaker commented on Jan 21, 2025

peerless2012 commented on Jan 21, 2025

moi15moi commented on Mar 11, 2025

icbaker commented on Mar 12, 2025

peerless2012 commented on Mar 12, 2025

e1ker commented on Mar 13, 2025

peerless2012 commented on Mar 14, 2025

GlassedSilver commented on Mar 14, 2025

nigelmegitt commented on Mar 14, 2025

GlassedSilver commented on Mar 14, 2025

kayodde commented on Sep 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions