(mediaElement & captionator) in videoPlayer

[Clark, Colin]

Hey Alex,

Great research, and very helpful summary! Some comments below...

On 2012-01-24, at 3:07 PM, Novak, Alexey wrote:
> Few notes:
> 	• It seems like we are going to use mediaElement only in a case of old IE versions OR maybe some old versions of Safari, Opera ?

We'll aim to use feature detection rather than user agent detection, so it should work on any browser that doesn't support <audio> and <video>. That said, we don't actively support really old versions of Safari or Opera, so in practice we'll be supporting MediaElement.js on IE6, 7, and 8, I think.

> 	• Also, I have not included different file versions into this simple diagram since during the test was using only with .mp4 file. Not sure what file formats we are going to support and how far would we would go in order to support them. Just in case, a matrix of supported formats by mediaElement could be a point of interest for us: http://mediaelementjs.com/ , take a look into Browser and Device support section.

The short version of codec and container support across browsers is roughly this:

* Safari and IE 9+ exclusively support H.264 encoded video in an MPEG-4 container
* Chrome, Firefox, and Opera exclusively support VP8 video in a WebM container

Typically, authors are asked to create two different versions of their videos, one encoded with H.264 and another for WebM. Until Apple is willing to accept a royalty-free codec, this will unfortunately be the case. In the meantime, services like vid.ly help.

After we meet our minimum viable user experience for the Video Player component, I think we should consider dynamically loading either Broadway.js (https://github.com/mbebenita/Broadway) or Route9.js (http://badassjs.com/post/13551173773/route9-js-a-vp8-webm-decoder-in-javascript) in cases where we encounter a video without the appropriate format for the user's browser. These are JavaScript-based decoders for H.264 and WebM, allowing us to decode video formats (at a higher performance cost) even when they aren't supported natively by the browser.

Given how inaccessible and unpopular Flash is, we want to fall back to MediaElement.js's Flash alternative only when we absolutely have to.

> 	• It looks like Captionator is using WebVTT formats so we might need to do some extra work if we want to parse JSON files as captions in videoPlayer.

For the time being, I think we should deprecate our use of JSON captions. While I am not all that excited by the design of WebVTT as a caption format, our JSON format was really just a quick sketch. We'd have a long way to go to create a better caption format and maintain a bi-drectional conversion utility. At this point, we're probably better of working with the WebVTT group to influence its direction, rather than making our own weird format.

> 	• It looks like mediaElement is using SRT formats for its captions. But SRT files are usually very close in format to WebVTT so it would be interesting to see if mediaElement can play VTT files instead of SRT.

It's my understanding that they're very slightly syntactically different, and require a different parser as a result. Don't get me started with "if they'd just used JSON, we wouldn't even need a parser..." ;)

It's worth a shot, but I gather that WebVTT and SRT are close but not interchangeable. Ideally, I'd like to see us port Antonio's "Rosetta Stone" caption converter from PHP into JavaScript where it can run either in the browser or on the server using Node.js. Then we should be able to easily handle any format out there.

> 	• Captionator will block any click event on the videoPlayer since it is been rendered as a DIV element on top of a HTML5 video component capturing any click events. I will try to find the way to ask a question about passing a click event through on a bug tracker if there is one OR will ask creators of Captionator directly.

Is it not possible to customize the appearance and rendering of Captionator's captions? If not, that might be something we'll need to fix and contribute back.

Perhaps it's worth doing a more in-depth evaluation of the library, looking at it through the lens of Infusion's core values: stylability, DOM agnosticism, customizable, and mashup-friendliness. Michelle might be a good person to pair with for this task.

> Just a question out of curiosity: what is the main strength of our player in comparison to someone who would pick mediaElement with all its features and strength combined with captionator? 

The Studios page describes the unique features of our Video Player component pretty well:

http://studios.fluidproject.org/?p=1

In terms of technology, you can see the notes from an evaluation we did of most HTML5 video players available at the time:

http://wiki.fluidproject.org/display/fluid/Evaluations+of+Existing+HTML5+Video+Players

You'll see a common theme amongst many of them. They're all built in a "take it or leave it" way--big, monolithic feature sets that can't be easily customized or adapted for different types of content, different applications, etc. Over and over again we found that developers hard-coded their markup and other presentational aspects, making it hard for users to deeply skin and customize them.

In short, we want a video player that has a lot of rich layers to it, beyond just captions--synchronized audio descriptions, transcripts, etc. We want users to be able to state their preferences for video once, and have the Video Player automatically adapt to those preferences, wherever it's located. So configurability becomes a huge requirement for us.

Hope this helps,

Colin

---
Colin Clark
Lead Software Architect,
Inclusive Design Research Centre, OCAD University
http://inclusivedesign.ca