Image Graph is powering the new Bing Image Search experience

Image Credit: Bing Blog

Earlier this month the Bing Image Search team implemented new changes to the way image search is done in Bing. The new changes brought a responsive design to exploration as well as shifting information to more accessible areas. Bing’s new Image Search is also tweaking its Image Graph to surface contextual information right on the search page, in an attempt to lessen the travel users make when searching for images and subsequent information.

Apparently this development is an arduous task and, Dr. Jan Pedersen, Chief Scientist for Bing and Information Platform R&D, and a couple of her colleagues Arun Sacheti and Eason Wang provided us a bit of insight into how the new Image Graph works. In a very lengthy blog post from the team, we come to find out that Image Graph is a nifty algorithm of sorts. Image Graph does a lot of the heavy lifting when it comes to sorting out URL from distinct Image information. The Image team found that around 59% of images have a least one duplicate on the internet. In most cases, images have up to thousands of copies all with distinct URLs. These copies mean each image on a page has a unique document text and query associated with it while only offering the smallest alterations to visual differences on information about the image. Traditionally users would have to sift through and analyze many of these web documents to find the image content they may have been initially searching for. With Image Graph, that process becomes more automated.

Image Credit: Bing Blog

Image graph primarily searches the internet for all duplicated images. Once they are collected, the graph then sorts the information on or about the image. After that, the graph then clusters similar images and all their associated metadata together to save users the time and present the most relevant image possible. Now that the team is working heavily with Microsoft Research they have been able to build a more efficient and scalable graph model with 100s of billions of nodes and counting. Surfacing the right image was only part of the graph’s goal, the other was to link content with images.

  • 1.Key concepts within the image through state of the art image understanding capabilities developed by Microsoft Research and Bing
  • 2.Context around the image within the Web page where the image remains hosted on the web
  • 3.High volume user interaction with the content
  • 4.Knowledge received directly from partners through annotated feeds

According to the team, “Our goal is to help users be inspired, learn more and do more with image search. Our team mines through each of these sources to generate datasets in billions of content clusters to support each of these user goals. Examples include Best Representative Query (BRQ), Captions, Related Collections, Shopping Sources and More Sizes.”

Image Credit: Bing Blog

The Image Graph is also utilizing curation and social awareness to bring like-minded or similar collections of images together for users. The graph is now bringing the collection of images from other sites like Pinterest, Tumblr or Etsy into search. The images will remain independent of each other. However, if someone happens to look up “living room decorating ideas” then the Image Graph will offer a collection of images from others who have similar styles, taste or preferences as well. The clustering of images occurs once an image is searched or selected and extends to colors, furniture, or design.

Image Credit: Bing Blog

Also new is the positioning of the Best Representative Query or BRQ. The information tag that used to be at the bottom corner of images would only appear when the image was hovered over. Now that information tag is present and in full detail at the bottom portion of the search panel. The new Image Graph can also mine text from images and produce descriptive Image captions. But how does the team determine what is useful and what isn’t? They’re glad you asked.

We defined ideal captions as ones that would:

•Help with understanding the image

•Be interesting enough to stimulate curiosity

•Identify key concepts within the image

•Be grammatically correct

New in image search is the added features of built-in shopping guides and tools. The Image Graph once again does some heavy lifting to connect the dots for users to ideally, keep them out of e-commerce hell when it comes to online shopping. “We regularly index shopping pages across the web, pages that can be classified as shopping pages based on content within the page as well as content present in structured tags based on OpenGraph,, RDFa and Microdata/Microformats. Based on the image search graph, we can find various shopping sites selling the same product by flagging where the same image content appears in different shopping pages,” explains the team.

Image Credit: Bing Blog

Lastly, the Image Graph is enabling reverse image lookup. So instead of dealing with the inconvenience of opening up several image search sessions or losing one’s image while, on discovery, image search allows for reverse look up. The reverse lookup of images happens in near-time through the Bing Image Match extension. The extension acts a history of image searches done by the Image graph and presented to users without a user having to search themselves.

The Image Graph has been a moderate evolution in Bing’s search history. The evolution began from URL graph, to content graph, to knowledge graph and now Image Graph looks to build on that foundation and spread throughout the rest of Bing and other Microsoft experiences. Hopefully, this means great things for services like Cortana, Xbox, Azure and Office 365 in the near future.

Share This
Further reading: , ,