alexandercerutti / sub37 Goto Github PK

With the current implementation, subtitles serving might not honor the media's playback rate and therefore go out of sync.

A solution should be evaluated so that every adapter can benefit from this change.

Interval Binary Tree should probably lookup like with a multiplier to get the right amount of cues on the same interval of time or whatever

As per the first version of sub37, the whole architecture expects Adapters always to return a ParseResult, containing data and perhaps some ParseError.

The idea was to have ParseResult and ParseError exposed by the BaseAdapter so that every adapter could use them without importing anything else and without making @sub37/server export other material.

However, the current situation reveals itself to be somehow inconsistent:

1. BaseAdapter/index.ts exports BaseAdapter (default), ParseResult, and ParseError classes. We should technically move Parse* to a different file;

2. BaseAdapter exposes ParseResult as an instance property, but not ParseError;

3. Related to the points above, tests and WebVTT adapter use ParseResult from there, while BaseAdapter should have been the entry point to access them.
   This happens because BaseAdapter.ParseResult is actually a method that returns a new ParseResult and, therefore it cannot be used as right-hand in an instanceof comparison: we must use instanceof ParseResult, but we should use instanceof BaseAdapter.ParseResult;

4. BaseAdapter exposes both ParseResult and ParseError types by using a namespace;

5. ParseError class is exposed from BaseAdapter/index.ts module, but it is not used as a class (this happens due to the point 3, so it is an useless class.

Fixing the whole situation is quite breaking, so should be fixed in the next major.

According to the image below, the Youtube-like rendering mode (which consists of shifting down the first line, when the mode is enabled), shifts down the content but maintains an empty line (which is utterly ugly and wrong).

This probably happens because translateY is applied to the wrong element.

To enable the Youtube-like mode, sample's script.mjs should be edited to add the following lines on the bottom of the file.

/**
 * @type {CaptionsRenderer}
 */

const rendererElement = document.getElementsByTagName("captions-renderer")[0];
rendererElement.setRegionProperties({
	shiftDownFirstLine: true,
});

With the integration of the TTML adapter, we are evaluating if we should support SMPTE ttp:timeBase.

There's also an extension of TTML which supports some features.
The article on Speechpad (linked below) describes them nicely.

Notes on adapters

Should we support SMPTE-TT inside TTML Adapter or in a different adapter?

If choosing a different adapter, we put ourselves in front of a choice: should we duplicate the adapter code or make adapters extendable?

When choosing the extendability, we are talking about changing the architecture to let adapters be hookable or something like that. Extenders should be able to intercept the parsing phase. Intercepting points must be defined.

Notes on captions-renderer

Of course, this change will go to impact also the captions-renderer. This is due to the support of bitmap images by SMPTE-TT, along with rendering style.

Useful resources

• https://ieeexplore.ieee.org/document/7290363
• https://www.speechpad.com/captions/smpte-tt

When styling global cues (like with WebVTT), the second row of a cue does not preserve the same style.

Minimum repro case

WEBVTT

STYLE
::cue {
  color: peachpuff;
}

00:00:00.000 --> 00:00:20.000 
Soffro anche un tantino
di sindrome di Tourette.

Result

When a document is well-composed (no unordered cues, correct timings, etc.) or perhaps streamed (during live content), the Interval Binary Tree beneath the Tracks, will be an all-right tree. This might lead, potentially, to higher retrieving time to reach the last element to show as the current time increases (as we have to navigate the whole tree on the right - e.g. when the content is in the middle of its duration).

The solution is to let it auto-balance, which is an AVL-Tree feature (it can be seen here) when new cues are available. This would let us cut in half the time to retrieve the time of retrieving.

Of course, the discussion is open (if anyone will ever read this 😄) and to implement an auto-balancing, we should decide first a limit after which the balancing is needed.

A little concern that comes to my mind is that auto-balancing could be decided to be performed in as small as possible chunks, as it might require some time (depending on the number of cues in the tree). Even if it is decided to be performed as a micro-queue activity, it should be performed as fast as possible to not block the retrieving of the current cues, which happens through a setTimeout (hence "macro-queue" or whatever is it called).

The vertical-writing mode is not supported in the current situation, either in WEBVTT or the upcoming TTML Renderer. In order to support it, we should change the CaptionsRenderer in order to paint them.

Also, we should change the RenderingModifiers in order to support eventual writing modifiers.

For what concerns WEBVTT, these are for sure the affected rows:

In order to support vertical writing, this resource might be considered valid. It contains W3C References for CSS, to which we might translate RenderingModifiers: https://w3c.github.io/i18n-drafts/articles/vertical-text/index.en

When using a non-targeting (valid but no matching element) CSS selector, all the elements get that style mistakenly applied on.

In the repro case below, the selector used in the STYLE is wrong as it should be v[voice="Fred"].

When using this selector, also cues containing <v Bill> become blue. None should become blue.

Minimal repro case

WEBVTT

REGION
id:fred
width:40%
lines:3
regionanchor:0%,100%
viewportanchor:10%,90%
scroll:up

REGION
id:bill
width:40%
lines:3
regionanchor:100%,100%
viewportanchor:90%,90%
scroll:up

STYLE
::cue([v="Fred"]) {
  color: blue;
}

00:00:00.000 --> 00:00:20.000 region:fred align:left
<v Fred>Hi, my name is Fred

00:00:02.500 --> 00:00:22.500 region:bill align:right
<v Bill>Hi, I’m Bill

00:00:05.000 --> 00:00:25.000 region:fred align:left
<v Fred>Would
<00:00:05.250>you
<00:00:05.500>like
<00:00:05.750>to
<00:00:06.000>get
<00:00:06.250>a
<00:00:06.500>coffee?

TTML adapter is still in the working, but I can surely create the issues for the features that won't be shipped with the first version of it.

tts:fontSize supports providing two <length>. According to the conversion table on the standard page, TTML

adds optional second length (or percentage) for specifying separate horizontal and vertical scaling of glyph's EM square;

However, to achieve such vertical scaling (perhaps horizontal? The opposite dimention of font-size), we need to change how the renderer works.

Specifically, along with font-size, we need to emit a transform: scaleX()/scaleY(), with the appropriate scaling factor, and apply it on an element that owns display: inline-block (hence, three properties to emit).

This translates, for the renderer, that it should create a new depth-level, a new span or whatever, that will let it specify such styles without impacting the previous or past sections.