Many forms of paper documents are utilized in our daily life and business, and adding to documents such markings as QR codesNote1 that can be extracted to provide related information and services has recently become prevalent. However, those markings must be added when creating a document, given the difficulty of adding markings to already prepared documents due to a lack of space and obtrusive visuals. To address this challenge, Fuji Xerox has developed nearly transparent markers (called Embedded Media Markers) using technology to retrieve similar images. The creator of the document can add this marker after the document has been created, without having to change its content or layout.
These nearly transparent Embedded Media Markers indicate and provide a link to related digital content. For instance, by using a mobile phone camera to capture an Embedded Media Marker on a paper document such as a flier, you can display the linked digital content on the mobile phone. This is made possible by extracting the image features from the area printed with an Embedded Media Marker and searching for the digital content linked to those features.
Fig. 1 illustrates the flow of creating an Embedded Media Marker. First, an authoring tool developed for this technology is used to specify the area in a document in which to embed the information and digital content (e.g., video, still image, text) to be linked. Then, after the authoring tool confirms the proper size and location for extracting the image features, a nearly transparent marker is created in the area. The image data encompassed by the marker boundary and the linked digital content are sent to a server together, and then the features of the image inside the marker and the digital content are registered in a database. The marker indicating where the digital content is embedded is also printed on the paper document. In this way, links to related information can be added to fliers or catalogs that have already been made.
Fig. 1: Flow of creating Embedded Media Markers
Fig. 2 illustrates how a smart phone detects the marker. By capturing the marker area with a smart phone camera, the image features in the marker are calculated. According to those features, the digital content in the database of a server linked to the marker is identified and sent to the phone.
Fig. 2: Using a smart phone to detect Embedded Media Markers
In order to accurately identify the digital content sent to the user's phone, markers must be robust to diverse photographing conditions (e.g., capturing range, rotation, shifting, partial occlusion, blurs) so that the features can be calculated efficiently. To make this possible, Embedded Media Markers employ a method based on local features that allocates multiple local features to keypoints in an image. First, in the same way as other existing methods based on local features, a keypointNote2 is determined according to the scale-space extrema of Gaussian derivatives. Then, from a scale higher than where the keypoint was detected, four other points are selected according to a rule (Fig. 3-A), and all five points (including the original keypoint) are used to calculate eight-dimensional gradient information (Fig. 3-B). The eight-dimensional gradient information of the five points is then combined to create a 40-dimensional vector that is considered a local feature of the keypoint (Fig. 3-C). The set of local features for each keypoint on the image becomes the features of the image.
By registering in a database both the features extracted from the image inside a marker and the related digital content as a pair, and then searching the database for the same features as extracted from the image read by a smart phone, the digital content linked to the marker can be identified.
Fig. 3: Local feature calculation