AdVis: Video Advertising by Content-based and Semantic-based Matching

and Ad Scheduling Optimization

Kuan-Ting Chen, Wei-Shing Liao, Winston H. Hsu

With the prevalence of recording devices and the ease of media sharing, consumers are embracing huge amounts of Internet videos. There arise the needs for effective video advertisement systems following their phenomenal success in text. For such strong demands, we propose a novel advertising system, AdVis, which automatically associates relevant ads by two promising methodologies – content-based and semantic-based visual matching. Analogous to AdWords (the prevailing text-base adverting system), the former specifies ad targets by characteristic images, referred to as adImages, and by high-level semantic concepts or events, referred to as adConcepts, in the latter case. AdVis avoids the pitfalls of poor tagging qualities in shared videos and provides a brand-new venue to specify ad targets by visual matching. Meanwhile, the rich contextual video constituents along the temporal dimension are exploited for advertising as well. To maximize the ad system revenues and user perception, we formulate the visual matching scores and the parameterized bidding information as an optimization problem. Through preliminary experiments, we found the proposed visual matching methods invariant to certain distortions commonly observed in shared videos and the ad scheduling algorithm providing contextually relevant ads to the viewers as shown in subjective evaluations.
　

An example movie and its user-contributed tags, which do not mention those rich contextual constituents (e.g., characteristic images, high-level concepts, or events, etc.) and their occurring time points in the video. Conventional text-based advertising systems associate ads based on the tags or the title along with the video. On the contrary, the proposed AdVis system exploits those rich contexts in the videos through a content-based (adImage) or semantic-based (adConcept) visual matching. Advertises can associate their ad targets with such visual contexts (e.g., the appearance of “Adidas” logo, during a basketball game, a break in a sports video, etc.).  Moreover, an ad scheduling algorithm is considered to optimize system revenues and user perceptions.

• A novel advertising system which automatically associates relevant ads by content-based and semantic-based matching.

• A system can further avoid the pitfalls of poor tagging qualities commonly observed in shared videos.

• An ad scheduling algorithm for maximizing the system revenues and user perception for advertising in shared videos.

• Efficient indexing and effective matching methods for utilizing spatial contextual cues.

• An improvement for spatial verification in content-based object-level matching with an unsupervised learning method.

• This framework can be one of the promising application platforms to accommodate state-of-the-art researches for videos and photos.

   This system includes three major components:

1. Advertiser interface contains the advertiser-provided bid, daily budget, and ad video along with the adConcept or adImage.

2. Matching system is responsible for determining the (probabilistic) presence of those visual contexts specified by advertisers. The matching methods are classified into two categories – semantic-based and content-based; the former is via specifying adConcepts by concept and event detection modules applied over the shared video databases; the latter is via adImages by content-based matching.

3. Advertising system will select the ads to optimize the revenues and user perception at suitable ad insertion points by an online scheduling algorithm. Unlike AdWord algorithm, where keyword matching is deterministic, AdVis needs to consider a soft matching between adImages and adConcepts. Besides, videos have an extra dimension in time rather than a simple 2-dimensional layout as web pages or documents. We need to further consider the temporal alignment issues between the source videos and scheduled ad videos. Furthermore, instead of degrading user perception for viewing videos, we need to consider the contextual relevance for inserting ads.

Due to the explosion of videos and photos and the needs for effective manipulation for such large-scale multimedia data, semantic understating had been an active research for years. Marjory there emerge researches in semantic concept detection and recently the definition of multimedia concept ontology, or even sports event detection . Such emerging technologies provide great opportunities for semantic understanding in videos and are applicable in the proposed AdVis framework for specifying the contextual targets for video advertisements.

• Semantic Concept Detection : Example photos from Flickr with the automatic concept detection results by the Columbia374 detectors. Note that Columbia374 were originally trained by the corpus and annotations in a set of broadcast news videos.

• Sports Event Detection  : In tennis event detection, we adopted the method proposed in [7].  The sports events will be parts of the adConcepts for matching and scheduling later in the ad scheduling algorithm and server as contextual cues for advertisers to specify ad targets in videos. Here is an example of sport event detection result, game_break.

In content-based visual matching, the advertisers can specify ad context by the appearance of characteristic images, thereby adImages. Such characteristic images can be those related to a company (e.g., Puma logo), a product (e.g., a sneaker), or a location (e.g., Golden Gate Bridge), etc. Here we show some examples of matching result and spatial verification as below.

• Matching Results: Four examples of image matching results. In each example, the left side shows its AdImage (characteristic image objects) and the right side shows the matching frame. The range of matched feature points is denoted by the red rectangle in the matching frame and each matched pair is connected by lines between AdImage and the target image.

• Spatial Verification:Top-five retrieved images (1024 × 768) for the query image (216x178) in the top row of (a) traditional content-based visual matching and (b) the ones after applying spatial constraint.

• All 16 images used as adImages

Mets	NYPD	Pickle	Flower
FDNY	Sale	Puma	UnderAmour
Adidas	2 Hours	Police badge	Wedding Bells
P.C. Richard & Son		Brooklyn Towers
Happy Madison		Universal