Email this article 
Patents and their effect on Standards:
Open video codecs for HTML5
Silvia Pfeiffera
(a) Dr Silvia Pfeiffer is CEO at Vquence Pty Ltd Australia, works within the non-profit open source software organisation Xiph.Org, and has been a contractor for Mozilla on technical issues around video for the Web
Abstract
HTML5 is an updated version of the hypertext markup language that has been empowering the World Wide Web for the last 20 years. One of the things that HTML5 introduces is a <video> element, which make video content as simple to include into Web pages as images. Similar to the issues that had to be overcome with the introduction of the <img> tag in 1993, we are now facing the issue of a common baseline codec for the <video> element – a format that all browser vendors can implement from a rights point of view and will agree to implement. Ogg Theora/Vorbis has been proposed as a solution, but only Mozilla Firefox and Google Chrome have agreed to implement it. Right now, we are at an impasse as the patent situation around video codecs is unclear and different parties take a different stance. This article discusses the issues in more detail.
Keywords
Standards, patents, open source, open standards, submarine patents
Info
This item is part of the Tech Watch section of IFOSS L. Rev. For more information, please consult the relevant section policies statement.
Introduction
On February 28th, 2007, Opera proposed the introduction of a video element into HTML51. The <video> element, similar to its counterpart the <audio> element, aims at vastly simplifying the use of audio and video content on the World Wide Web. When implemented, there will not be a need to install plugins into your Web Browser of choice because it will support the <video> element out of the box. Such an element will further allow new forms of publishing of audio and video in a more integrated fashion with HTML. An example is the projection of a video on the surface of a cube which is being drawn in a <canvas> element. A whole new world of online applications is enabled when such elements exist.
A few weeks after the initial proposal of the <video> element, Opera CTO Håkon Wium Lie stated in a talk given at Google:
“I believe very strongly, that we need to agree on some kind of baseline video format if [the video element] is going to succeed. [...] We want a freely implementable open standard to hold the content we put out. That's why we developed the PNG image format. [...] PNG [...] came late to the party. Therefore I think it's important that from the beginning we think about this.”2
Thus was born the need for a common "baseline codec" for HTML5 video (audio always implied). A baseline codec is essentially a video (and audio) encoding format that is supported and implemented by all browser vendors.
Håkon further stated requirements for the video element as follows:
“It's important that the video format we choose can be supported by a wide range of devices and that it's royalty-free (RF). RF is a well-establish principle for W3C standards. The Ogg Theora format is a promising candidate which has been chosen by Wikipedia.”3
Ogg Theora is a codec developed by the Xiph.org Foundation as an open source codec4. Theora is a derivative of a codec that was earlier developed by On2 Technologies under the name VP35 and released as open source in September 20016. The VP3 source code was originally published on www.vp3.com. This site was closed down when the code and its further development were moved to Xiph.Org. On2 Technologies is currently in the process of being acquired by Google.
The license under which On2 Technologies published the VP3 source code reads as follows:
“On2 represents and warrants that it shall not assert any rights relating to infringement of On2's registered patents, nor initiate any litigation asserting such rights, against any person who, or entity which utilizes the On2 VP3 Codec Software, including any use, distribution, and sale of said Software; which make changes, modifications, and improvements in said Software; and to use, distribute, and sell said changes as well as applications for other fields of use.”7
This implies that On2 Technologies asserts to never pursue its patents on the Theora codebase being an improvement to the On2 VP3 Codec Software.
Further, Ogg Vorbis is an open source audio codec developed and published by Xiph.Org since about the year 2000. Vorbis was developed with a clear intention of only using techniques that were long out of patent protection. Vorbis has been in use by commercial applications for a decade now, including Microsoft software and many games.
After VP3 was published and turned into Theora, Ogg Theora with Vorbis as the audio track became the first unencumbered video/audio codec software.
What exactly is an “open codec”?
The term "open codec" has been used in at least two substantially different environments: "open standards" and "open source".
In the open source environment, a firm requirement of open software is that it is published under a license that satisfies the open source definition8. In particular it contains the following sentence:
"The license shall not require a royalty or other fee <...>"
Thus, an open codec does not only consist of open source software, but also requires a royalty free license such that the code is actually usable in many different circumstances.
In the open standards environment, the definition of "open standard" is not as clear-cut9. Mostly, people regard a specification that has been developed by a committee of representatives from multiple organisations and published by a standards organisation as an "open standard", independent of whether it costs to purchase the specification or whether an implementation requires payment of royalties.
The World Wide Web Consortium (W3C), which is the standards body that is also publishing HTML, is an exception amongst standards bodies in that it seeks to issue only Recommendations that can be implemented on a Royalty-Free (RF) basis10. Thus, an open standard developed by other standards bodies may not be regarded as being open enough for the W3C. This is the case for the MPEG-4 H.264/AVC codec, which has been identified as a candidate for a video codec11. H.264 has been approved as a standard jointly by the International Telecommunications Union (ITU) and the International Standards Organisation (ISO/IEC).
H.264 has a clear requirement for royalties to be paid on the large attached patent portfolio. At this point in time, publishers and distributors of H.264 content on the Internet are in a honeymoon phase where there are no royalties until 30th December 2010, but then, the royalty shall be “no more than the economic equivalent of royalties payable during the same time for free television”, which can reach up to $5million annually depending on audience size12.
While such licensing terms may seem reasonable and non-discriminatory (RAND) to ITU and ISO, many of the smaller Internet publishers, and in particular not-for-profit organisations, will find it hard to comply with such terms.
The honeymoon phase and the codec’s high quality have made sure that H.264 now has a major share in the video formats that are being distributed over the Internet. However, with such license terms associated, it is not an acceptable format as a baseline codec for the HTML5 <video> element.
HTML5 baseline codec requirements
Opera's first experimental build of the HTML5 <video> element used Ogg Theora and Vorbis as the video and audio codecs being inspired by Wikipedia's exclusive use of Ogg Theora/Vorbis. Consequently, the first specifications of video in the standard13 also included a suggestion of Ogg Theora/Vorbis as baseline codecs, with an expectation that this would encourage all browser vendors to make use of these open codecs:
“User agents should support Ogg Theora video and Ogg Vorbis audio, as well as the Ogg container format.”
However, by December 2007, it was clear to the editor of the HTML5 draft, Ian Hickson, that not all browser vendors were going to implement Ogg Theora and Vorbis support. He therefore replaced the should-requirement for Ogg Theora/Vorbis with the following:
“It would be helpful for interoperability if all browsers could support the same codecs. However, there are no known codecs that satisfy all the current players: we need a codec that is known to not require per-unit or per-distributor licensing, that is compatible with the open source development model, that is of sufficient quality as to be usable, and that is not an additional submarine patent risk for large companies. This is an ongoing issue and this section will be updated once more information is available.”14
This was supposed to bring the issue to the attention of the browser vendors and make them start solving it by clarifying the HTML5 requirements on a baseline codec. At that time, Ogg Theora was being criticised for its inferior quality over H.264, its lack of implementations on mobile devices, and its threat of submarine patents, which some vendors like Apple, Nokia, and later Microsoft used as an argument to not support it15.
So, Mozilla contributed some funding16 to have the core developers of Theora improve encoder quality, which has led to amazing progress - some of which still continues. With some currently deployed encoders – in particular with the YouTube encoder - Ogg Theora can now be regarded as almost on par with H.264 when it comes to video quality.
Further, initial implementations of Ogg Theora/Vorbis on mobile devices have emerged, proving the point that Ogg Theora is a much simpler codec than H.264 and therefore does not need as much special hardware support on small devices to make it usable. Many are still waiting for the day on which Ogg Theora/Vorbis video can be viewed on an iPhone or iPod, claiming these as the ultimate proof of portability. It is only a matter of market demand until a device vendor will step forward and offer Theora hardware decoding.
Theora now meets all of the requirements listed by the HTML5 editor bar the additional “submarine patent” risk. In contrast, the competing H.264 doesn’t meet the license requirements and certainly the existence of submarine patents on H.264 also cannot be completely excluded.
The term “submarine patent” is being used rather loosely in the HTML5 community in reference both to patents that have not been identified yet as being infringed by a technology, as well as patents that have only been registered so recently that they are virtually unpublished and can therefore not be identified as being infringed. The latter is the real meaning of a “submarine patent” according to Wikipedia17, but both are being referred to as “submarine patents” in HTML5.
The risk of the existence of “submarine” patents on Theora is potentially slightly larger than on H.264, since many of the patent developing companies on video codecs were part of the H.264 development process and have their patents entered into the H.264 patent pool. The video patent developing companies, however, tend not to take part in W3C standardisation activities and therefore their patents are not covered by W3C. Thus, Theora supposedly poses a larger threat than any MPEG codec.
On the other hand, those patents are known and already licensed to Apple, Microsoft etc, so they would not pose additional risks to existing license holders. The number of as yet unknown “submarine” patents threatening Theora is possibly roughly the same as the ones threatening H.264. It is this risk, though, that continues to hold back Apple and Microsoft (and possibly others) from implementing Theora support in their hardware and software.
The current HTML5 video codec situation
With a goal of taking HTML5 to a Last Call status by the end of 2009, the HTML5 editor, Ian Hickson, had to clean out controversial aspects from the document. Seeing no progress on the baseline codec decision, on 29th June 2009, he removed the section on baseline codecs from the HTML5 specification altogether18.
By then, the situation had gotten worse: video element support had been implemented in Mozilla Firefox, Apple Safari/Webkit, Google Chrome, and Opera, but each browser vendor had done their own analysis of the situation at hand and different baseline codecs had been chosen. While Mozilla and Opera only supported Ogg Theora/Vorbis, Google decided to support both, Ogg Theora/Vorbis and H.264, and Apple decided to support only H.26419.
Mozilla shipped Ogg Theora/Vorbis with their release of Firefox 3.5 in June 2009 after having undertaken their own patent research and risk analysis20. Unfortunately, the outcome of this research has not been published.
Google also did their own patent research, which Chris De Bona outlined in a post to the HTML5 mailing list21 on "Google's use of FFmpeg in Chromium and Chrome". FFmpeg in Chrome supports both, Ogg Theora/Vorbis and H.264. This is not a problem for Ogg Theora/Vorbis. However, it has been identified in that email thread that it seems to be a problem for people wanting video element support, but not having acquired their own license for H.264 support. It seems, Google themselves are protected by their license of H.264, but their users aren’t. To the interested reader with a legal background, that email thread may be an informative read.
Opera continue development of Ogg Theora support and are shying away from H.264 for its license costs22. Their currently released browser does not support the video element, but it is expected that the next release will support it with Ogg Theora as the only codec23.
Microsoft is a “fan” of the audio and video elements24, but have early on in the discussion about baseline codecs stated that “Our legal people do not believe that Ogg Theora has much of any chance of being free from patent issues”25. It seems they share the views of Apple and Nokia in this respect. They haven't spoken up about what baseline codec Internet Explorer is planning to support.
Apple asserts that Theora holds a larger “submarine” patent risk than any of the video codecs that have come through standards bodies and therefore refuses to implement native support in its applications and hardware devices for it26. Apple has concerns because of issues raised by Nokia's video patent expert27 and because the license that Xiph received from On2 may not apply to third party implementations of Theora28. Nobody knows what the recent $106M acquisition of On2 Technologies by Google29 may change in this situation. Google has not made any statements about the codecs they are acquiring and what their intentions are.
Further, Apple is worried about the additions that Xiph.Org made to VP3 to turn it into the current version of Theora, since these could be covered by real submarine patents30. The changes are all publicly listed31 and consist to a large extent of bug fixes, though some more interesting extensions have been contributed. A patent analysis of the major changes has not been published.
Apple also claims that because video codecs influence all of Apple's business, it would be a big target for patent trolls, while Mozilla's Firefox and Google's Chrome businesses are both too small for a patent troll to be encouraged to step forward at this stage. Now, if YouTube converted from H.264 to Ogg Theora that would be a different situation. Interestingly, several other video hosting sites, amongst them Dailymotion32, Wikimedia33, and Archive.org34 already support Ogg Theora/Vorbis and many smaller community video sites have also decided to go with this simple and usable solution.
Apple would prefer for Theora and Vorbis to be ratified by a standards body with involvement from the main video and audio codec patent holders. However, such a process would introduce so much delay as to exclude Theora and Vorbis as codec solutions in HTML5.
Unfortunately, the W3C is not able to employ lawyers to undertake an independent patent assessment for any codec, to expose “submarine” patents. However, the W3C could issue a call for a contribution of baseline codecs and go through a short but formal selection process, which would require patent holders to step forward. This would require inclusion of other standards bodies to ascertain that everyone had the chance to speak up in due time. We will see if such a process is possible and if indeed it will satisfy Apple, Nokia and Microsoft.
In the meantime, open source software developers continue to bridge technology gaps. It is currently possible to run Ogg Theora/Vorbis content in every browser using the Cortado35 Java applet. This is a similar approach to a Flash plugin with the advantage that the applet is loaded automatically and there is no installation process. A better solution is available for Safari/Webkit with a QuickTime plugin called XiphQT36. Once installed, Ogg Theora/Vorbis will be supported in Safari/Webkit. For Internet Explorer, an ActiveX control is in development that will also be usable in the same way.
Conclusion
It seems that in the near future, all technical challenges for universal support of Ogg Theora/Vorbis as a baseline codec for HTML5 video and audio will be solved – either through native browser support or through extensions. Whether the legal situation will be solved in parallel is doubtful. It is, however, clear that unless the H.264 patent holders radically change their stance on licensing, H.264 will not be a viable contender as a royalty free baseline codec for HTML5.
Further changes to the situation could come from currently proprietary closed source codecs - e.g. if On2/Google contributed a new codec such as VP8 to the discussion by open sourcing it and making its patents available under a royalty free license – or further even indemnify all users of the codec from and against any and all liability.
In the meantime, the HTML5 editor is still waiting for the browser vendors to come to an agreement on a baseline codec that satisfies the requirements. Until such a time, the patent situation with video codecs continues to hold back the standardisation and broad uptake of a fundamental section of HTML5.
Disclaimer
The views expressed in this article are my personal views, not my company's, Xiph.Org’s or Mozilla’s. I am not a lawyer and this is not a legal analysis. It is only provided to give a background on the situation of the HTML5 standardisation work for video.
Dr Silvia Pfeiffer is CEO at Vquence Pty Ltd Australia, works within the non-profit open source software organisation Xiph.Org, and has been a contractor for Mozilla on technical issues around video for the Web.
Licence and Attribution
This paper was published in the International Free and Open Source Software Law Review, Volume 1, Issue 2 (December 2009). It originally appeared online at http://www.ifosslr.org.
This article should be cited as follows:
Pfeiffer, Dr Silvia (2009) 'Patents and their effect on Standards:
Open video codecs for HTML5', IFOSS L. Rev., 1(2), pp 131 – 138
DOI: 10.5033/ifosslr.v1i2.21
Copyright © 2010 Dr Silvia Pfeiffer.
This article is licensed under a Creative Commons UK (England and Wales) 2.0 licence, no derivative works, attribution, CC-BY-ND.
As a special exception, the author expressly permits faithful translations of the entire document into any language, provided that the resulting translation (which may include an attribution to the translator) is shared alike. This paragraph is part of the paper, and must be included when copying or translating the paper.
1See email to the WHATWG mailing list by Opera dated 28th February 2007, http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2007-February/009702.html
2See video of Håkon Wium Lie’s Google talk at http://video.google.com/videoplay?docid=5545573096553082541&ei=LV6hSaz0JpbA2AKh4OyPDg&hl=un
3See Håkon Wium Lie’s page on the need for a video element at http://people.opera.com/howcome/2007/video/
4See Xiph.Org’s Website on Theora at http://theora.org/
5See On2 Technologies’ press release on the 24th June 2002 at http://www.on2.com/index.php?id=486&news_id=313
6See On2 Technologies’ press release on the 7th September 2001 at http://www.on2.com/index.php?id=486&news_id=364
7See Xiph.Org source code repository at http://svn.xiph.org/trunk/theora/LICENSE
8See Open Source Definition at http://en.wikipedia.org/wiki/Open_Source_Definition
9See Open Standard Definition at http://en.wikipedia.org/wiki/Open_standard
10See W3C RF requirements at http://www.w3.org/Consortium/Patent-Policy-20030520.html#sec-Licensing
11See W3C HTML Working Group Issue tracker, Issue #7 at http://www.w3.org/html/wg/tracker/issues/7
12See MPEG LA (License Authority) publication at http://www.mpegla.com/main/programs/AVC/Documents/AVC_TermsSummary.pdf
13See Archive.org’s June 2007 version of the HTML5 specification at http://web.archive.org/web/20070629025435/http://www.w3.org/html/wg/html5/#video0
14See Ian Hickson’s email in December 2007 to the WHATWG at http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2007-December/013135.html
15See as an example this story in Apple Insider http://www.appleinsider.com/articles/09/07/06/ogg_theora_h_264_and_the_html_5_browser_squabble.html
16See press release by Mozilla on 26th January 2009 at http://blog.mozilla.com/blog/2009/01/26/in-support-of-open-video/
17See Wikipedia article on Submarine patents at http://en.wikipedia.org/wiki/Submarine_patent
18See email of Ian Hickson to WHATWG at http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2009-June/020620.html
19See Opera blog post about video at http://my.opera.com/ODIN/blog/video-on-the-web
20See Mozilla press release at http://www.mozilla.com/en-US/firefox/3.5/releasenotes/
21See Chris di Bona’s email to WHATWG at http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2009-June/020035.html
22See email to WHATWG at http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2009-June/020620.html
23See tweet from Opera video element developer http://twitter.com/foolip/status/6923376494
24See email to W3C as discussed and linked at http://www.osnews.com/story/22182
25See W3C HTML Working Group Issue tracker, Issue #7 at http://www.w3.org/html/wg/tracker/issues/7
26See W3C HTML Working Group Issue tracker, Issue #7 at http://www.w3.org/html/wg/tracker/issues/7
27See Nokia submission to a W3C workshop on video for the Web at http://www.w3.org/2007/08/video/positions/Nokia.pdf
28See email from Apple to Xiph.Org at http://lists.xiph.org/pipermail/theora/2009-July/002415.html
29See discussion of On2 Technologies acquisition by Google at http://www.techcrunch.com/2009/08/05/google-acquires-video-compression-technology-company-on2-for-106-million/
30See email from Apple to Xiph.Org at http://lists.xiph.org/pipermail/theora/2009-July/002415.html
31See Xiph.Org version control system at http://svn.xiph.org/trunk/theora/CHANGES
32See Wikipedia article on Dailymotion http://openvideo.dailymotion.com/en
33See Wikipedia article on Wikimedia http://commons.wikimedia.org/wiki/Category:Video
34See Wikipedia article on Archive.Org http://www.archive.org/details/movies
35See Xiph.Org site at http://www.theora.org/cortado/
36See Xiph.Org site at http://www.xiph.org/quicktime/