Comic Book Style Video Summarization Technology (Manga)

As camera-equipped devices such as smartphones become more popular, camcorders are no longer the only option for shooting video. Today, we have more opportunities to shoot video, thereby producing a large video collection on storage devices. However, it is not easy to quickly find a specific scene in a video file from among many video files. In order to solve this problem, FX Palo Alto Laboratory, Inc., located in California's Silicon Valley, has developed a comic book style video summarization technology named Manga. This technology displays the content of a video file in a sequence of size-adjusted keyframes, in order to reflect the importance of representative video segments.

Fig. 1: Flow of packing a sequence of differently sized keyframes on a single page to create a video summary

First, a video is segmented based on its content. The importance score of each segment is computed based on the segment's length and image features. The segments with higher importance scores are then selected for inclusion in the final summary. A representative frame is extracted as a keyframe from each of the selected segments, and each keyframe is sized according to the importance score of the originating segment. Finally, the extracted keyframes are laid out in a specified space. The sizes of some keyframes may need adjustment to fill the space while maintaining the temporal order (Fig. 1). In this way, the content of a video file can be quickly browsed as a sequence of differently sized keyframes that reflect their importance. The visual summary allows users to easily locate a specific scene of interest and start playing from that scene by clicking the corresponding keyframe.

Related information

Case example