Visual search is an important new business tool that is changing the way people interact with E-commerce and social media platforms. Instead of entering a text query, visual search engines enable users to identify and locate items via photos snapped with their smartphone cameras. The rise of visual search in online stores is fueled by customers’ desire to discover new products and brands.

In this report, we use an Alibaba case study to show how visual search can help E-commerce customers find appropriate products in a massive catalog. We then examine how Pinterest uses the technology as an ad platform and to match brands and products to users. Finally, we look at how retailers can leverage visual search on their own platforms.

Product Discovery — Case Study of Alibaba’s Visual Search Engine

Alibaba’s Pailitao shows the value of visual search in E-commerce. It helps customers to find the apparel they want in a massive product catalog and enables AliExpress to better serve its global users.

Alibaba launched its visual search engine Pailitao in 2014, and by last summer it was averaging over ten million daily unique visitors. Pailitao users can search with an image and find matching products from among the billions of items on Taobao, Alibaba’s online shopping website. Pailitao covers almost all Taobao product categories, including fashion apparel, shoes, bags, furniture, makeup, and food. For example in fashion apparel — one of Taobao’s Top 20 categories — customers often use Pailitao to search for clothing or accessories worn by celebrities or influencers in photos. Pailitao greatly improves the user experience by eliminating the need to describe the target item using terms relating to color, style, brand, etc.

Pailitao quickly expanded beyond Taobao to integrate AliExpress, Alibaba’s international online retail service. Pailitao is especially suitable for AliExpress because it largely eliminates the language barrier for international searches. By enabling images as product search inputs, AliExpress can worry less about the translation of product descriptions and search queries in foreign-language markets, as these customers can find what they’re looking for using only photos.

Pailitao has also been outsourced. Samsung partnered with Alibaba to integrate the technology into its smartphone cameras and virtual assistant Bixby, enabling users to search for products by simply pointing their smartphone camera at an object. Video platform Youku also partnered with Alibaba to enable their audiences to directly search for and conveniently purchase products seen in videos.

Pailitao Integration in Youku (from Alibaba Yunqi community blog)

Pinterest case study — Ad Platform Powered by Visual Search

Pinterest’s Visual lens shows the value that visual search can bring to a social media platform — users can discover new brands that match their taste, and retailers can identify their target customers.

Pinterest launched its visual search engine Pinterest Visual Lens in February 2017. Users can take a picture of a product in the physical world and search for the relevant content (Pins) in Pinterest. By February 2018 Pinterest users were making some 600 million Visual Lens searches every month.

Use Pinterest Visual Lens to Search for Fashion Products

Visual Lens’ success is largely driven by users’ desire to discover new looks, brands and products. A Mary Meeker Internet Trends report found that 55 percent of Pinterest users want to discover and shop for new products — compared to only 14 percent for Facebook and Instagram.

Visual Lens excels at discovering relevant products which users are unfamiliar with or don’t know how to accurately describe. Pinterest says 97 percent of its search queries do not contain a specific brand name. Visual Lens automatically uses color, pattern, texture, and other information in a picture to search. Pinterest has also developed features for specific use cases. “Shop the Look” for example allows users to search with influencers’ photos and find relevant product pages directly in Pinterest.

Pinterest’s Visual Lens can help brands boost their product pages when users search with images. Companies can also gain understanding of their customers’ tastes through their searches. Popular Pinterest Visual Lens search categories include fashion apparel, home decoration, art, and food.

Home improvement supply retailer Home Depot, department store Target and fashion brand Tommy Hilfiger are all committed to Pinterest ads; and a growing number of brands are finding ways to leverage the new visual search feature. The platform’s mobile ads are a major force as about 80 percent of Pinterest monthly active users use the mobile app. With all the new business, Pinterest expects to double its revenue to US$1 billion this year and hit a total valuation of US$15 billion. The company is expected to go public in 2019.

How to Start Visual Search from Scratch

The potential business impact of visual search in E-commerce is tremendous. An investment in visual search-enabled social platforms can help emerging fashion and lifestyle brands identify their target audiences based on tastes, and convert them into customers. For established E-Commerce retailers, visual search can help customers quickly locate products from a massive catalog and remove language barriers.

For retailers who want to get started with visual search, many vendors provide customized visual search solutions to simplify implementation. Pinterest itself is actively working with external partners. Clarifai provides a full suite of visual recognition and searches solutions, which helped the Architizer marketplace build their visual search tool for construction materials. Amazon AWS’s perception system Rekognition helps online retailer TheTake sell apparel and accessories seen in popular movies. Syte.ai is a fashion-oriented visual search solution provider whose clients include Samsung, Marks & Spencer, and Kohl’s.

By leveraging vendors’ solutions, retailers can quickly start experimenting with visual search tools on their own platforms.