Broadband TV Conference Part 2: How to Measure Streaming Video Quality

Introduction:

This second article on the 2014 Broadband TV Conference summarizes a presentation by OPTICOM’s CEO on streaming video quality measurements. We think that topic will be very important for many players in the OTT streaming video and connected TV markets. In particular, we believe it’ll be quite valuable for adaptive bit rate OTT and mobile video streaming providers, in order to measure and then attempt to improve the Quality of Experience (QoE) of their customers.

Perceptual Quality Measurement of OTT Streaming Video TV Services, Michael Keyhl, CEO of OPTICOM

How do you measure streaming video quality? Very few seem to have good metrics on video Quality of Experience (QoE) for viewers, even though it impacts many participants in the OTT, SD/HD video content delivery business. The stakeholders involved in QoE for video subscribers/consumers include: content provider, OTT provider, pay TV providers (cable, satellite, telco), network operators (especially for mobile video consumption), device makers, video codec providers and mobile apps companies including Internet videos in their apps.

Michael Keyhl, CEO of OPTICOM addressed this important topic in a very enlightening Broadband TV conference session. Germany-based OPTICOM develops algorithms for measuring video quality and licenses that technology to test equipment, video analytics and other  OEM partner companies.

Mr. Keyhl said that existing standardized video quality measurements barely suffice when considering OTT streaming video. Fundamentally, all traditional objective testing standards are based on analyzing short video sequences -of only a few seconds in length. The Mean Opinion Scores are quite low (below 5) for OTT video quality measured that way.  Michael said  that “snapshots of 10 second videos are inadequate to assess re-buffering and long term streaming behavior.” Hence, there’s a need for new type(s) of subjective testing methods and procedures.

In an attempt to greatly improve perceptual video testing standard for streaming video services (including ABR), OPTICOM created Perceptual Evaluation of Video Quality- Streaming (PEVQ-S).  It was described as an “advanced framework algorithm for full-reference picture quality analysis in video streaming environments.” The rules (but not implementation) have been standardized by ITU-T as J.247: Objective perceptual multimedia video quality measurement in the presence of a full reference. Related follow on work on video quality measurements is taking place in the Video Quality Experts Group (VQEG) which produces inputs to various ITU Study Groups for recommendations they’re developing.

As opposed to the lightweight “No Reference” video quality testing type, Full Reference testing is more processing intensive, but offers the highest accuracy and is standardized by ITU. It’s based on differential analysis – comparing the degraded video signal with the original source video with access to the reference/studio source video. OPTICOM‘s PEVQ/ITU-­T J.247 is the standard for Full Reference Video Quality Measurement as noted above.

The different types of Adaptive Bit Rate (ABR) video streaming methods are illustrated in the chart below. As you can see, there are many combinations and permutations for video quality measurements.

A diagram showing different streaming methods.
Image courtesy of OPTICOM.

Note: In Adaptive Bit rate (ABR) video streaming, the transmitted bit rate, resolution and other aspects of each “media segment” varies according to bandwidth and resources available at the client (receiving device). Video quality significantly depends on the client behavior, such as negotiating bit rate with the video server depending on dynamically allocated bandwidth, streaming protocol and re-buffering. The Media Presentation Description (MPD) is used to convey that information from client to server.


OPTICOM’s PEVQ was said to be validated for many different types of video- not just OTT ABR- using subjective testing. The validated video formats were based on ITU-R recommendation BT.500 – originally named “CRT TV Quality Testing (SD)” and ITU-T Recommendation P.910 – “Multimedia (QCIF, CIF, VGA) and IPTV (HD 720/1080) Testing.”

Based on a fundamental requirement analysis to understand adaptive streaming artifacts, the design of a novel test method was described. A four layer OTT quality model was presented with these four layers (top to bottom): Presentation, Transmission, Media Stream, Content.

Michael said an OTT Video Quality Measurement technique needs to have the following characteristics/attributes:

  • be related to content quality as a reference;
  • accurately score encoding and transcoding artifacts =Media Stream Quality;
  • measure and compare the picture quality for different frame sizes and frame rates = Media Stream/Transmission Quality;
  • continuously track the different bit rates and evaluate how smooth the video player is able to interact with the video server in a congested network= Transmission Quality;
  • take into account the player and endpoint device characteristics as well as the viewing environment = Presentation Quality.

The architecture of OPTICOM’s novel approach to measuring streaming video quality was said to be able to “overcome the limitations of standardized perceptual video metrics with regard to adaptive streaming of longer video sequences, while maintaining maximum backward compatibility (and thus accuracy) with ITU-T J.341/J.247 for short term analysis.”

An end-to-end functional block diagram of streaming video source/destination measurement using PEVQ-­S is shown in the illustration below.  The four OTT quality layers are shown at the bottom of the figure.

Block diagram showing possible quality of service impairments from source to sink.
Image Courtesy of OPTICOM

Conclusions: 

  1. There’s a clear need for a streaming video quality measurement (VQM) technique which accurately evaluates video subscriber/consumer QoE
  2. QoE concepts must be completely reformulated and we must reinterpret video quality in the context of multi-screen use scenarios.
  3. Currently, there is no standard for subjective and/or objective VQM of ABR video streaming
  4. PEVQ-­S is proposed to resolve that problem, based on advancing existing standards, while maintaining maximum backward compatibility and validated accuracy.
  5. PEVQ-­S is well suited to evaluate all 4 OTT Quality Layers (from bottom to top): Content, Media Stream, Transmission, and Presentation
  6. PEVQ-­S allows for analysis of common ABR protocols and formats, various video codecs at various bit rates. It can analyze video at different frame sized and frames per second.
  7. PEVQ-­S is licensed by OPTICON to leading OTT, Middleware, and Test & Measurement vendors.  It will soon be built into many such products. OPTICOM says they have over 100 licensed OEM customers.

OPTICOM’s Demo:

OPTICOM had a demo at the conference where they measured ABR video quality under various simulated reception conditions. We certainly could detect a difference in quality during different time periods of the stream. The quality of each of the video segments were measured and recorded.

We think that such measurements would be especially useful for mobile OTT video streaming where RF reception varies depending on the wireless subscribers location and physical environment.

End Note:

Time and space constraints do not permit me to highlight all the excellent sessions from this two day conference. Such a complete report is possible under a consulting arrangement. Please contact the author using the form below, if interested:

0 thoughts on “Broadband TV Conference Part 2: How to Measure Streaming Video Quality

  1. Thanks Alan for writing this great summary. Being able to quantify Quality of Experience becomes increasingly important to service providers as they move content from managed to unmanaged networks. As we will see in your next article, the home network, particularly when it is wireless, is one of the bigger challenges with regards to varying bandwidths.

  2. In Feb 2013, Amazon produced a video “HOW-TO: Best Practices for Measuring Performance of Streaming Video” intended for OTT video content providers. http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/Amazon-Explains-How-to-Measure-Streaming-Video-Performance-87648.aspx

    Key issue is whether those video providers used the quality measurements to improve their videos delivery method? I don’t notice much of an improvement in the last year. I think the quality will be a function of the video player.

    1. Good point, indeed! For streaming video, quality to a large portion has become a function of the video player interacting with the (congested) network. This leads to a paradigm shift – it’s no longer adequate just to ensure sending high quality at the head-end side, but the overall perceived Quality-of-Experience needs to be benchmarked, including all network effects. Maybe this mindset is not yet shared by all responsible parties?

      PEVQ-S was just launched this spring at MWC2014 and NABShow, so we are excited to see a widespread use starting in Q3, hopefully contibuting to more mature video streaming networks.

  3. One year on from these comments and it seems that a real-world case of benchmarking video quality threw up some interesting insights. The August version of the German CONNECT Magazine – http://www.connect.de/dsltest/ (a major tech consumer magazine within Germany) recently commissioned a test of German DSL Networks, including video performance. The report is in German language though, which I hope won’t be too much of an issue.

    This initiative included 1.7 Million PEVQ-S measurements of video streaming networks, YouTube and Amazon Instant Video, conducted by the German test company Zafaco. An interesting finding on testing various CDNs was that the fastest network access did not necessarily provide the highest perceived video quality, in fact sometimes the slower one was the best. Even more interesting was that an OTT scored higher than the network they pass over, indicating some optimisation in place for the streams used. This therefore proves a tool such as PEVQ-S can provide valuable insight for CDN optimization and OTT providers, where other network based analysis cannot. It has been exactly the same situation for mobile telephony where end-user QoE testing has been very much the norm for the past 15 years at least.

    If you are interested, there is also a link for further details how Zafaco did the benchmarking: http://www.zafaco.de/uploads/media/kyago_Whitepaper_Multi_Play_Benchmarking_Plattform.pdf

Leave a Reply

Your email address will not be published. Required fields are marked *

I accept that my given data and my IP address is sent to a server in the USA only for the purpose of spam prevention through the Akismet program.More information on Akismet and GDPR.

This site uses Akismet to reduce spam. Learn how your comment data is processed.