Published: Fri, September 14, 2018
IT | By Lester Massey

Facebook builds 'Rosetta' AI to understand text in images and videos

Facebook builds 'Rosetta' AI to understand text in images and videos

In a blog entry, Facebook designers say they have made a devoted AI apparatus, considered Rosetta, to peruse the content that shows up in images (and different pictures and video outlines) that are shared to Facebook and Instagram.

To handle this monumental task, the company has built a sophisticated artificial intelligence called Rosetta.

Rosetta extracts text from more than a billion public Facebook and Instagram images and video frames (in a variety of languages), daily and in real time, and inputs it into a text recognition model. In the first step, Rosetta detects rectangular regions that potentially contain text. It can add the search results, and it can also scan for the harmful content. It will analyze the images as well as it will use the historical data which is rather than the visual profile of the individual characters to understand the writing. Facebook said that the approach would enable the Rosetta which can recognize any type of words of any length which is even the ones that it was not exposed to during the training phase of development. In other words, Facebook developed an AI that can tell if a meme is offensive.


While Rosetta has been created to scan and read images and video frame, a particularly interesting use case of the technology is in scanning the memes, which appear frequently on the company's two platforms.

"The rapid growth of videos as a way to share content, the need to support many more languages, and the increasing number of ways in which people share content make text extraction from images and videos an exciting challenge that helps push the frontiers of computer vision research and applications", the company writes. It will be extended to more areas in the future.

Facebook says that Rosetta has already been widely adopted within various teams at the company and with Instagram, and noted that the text extracted from images is also being used in various machine learning models, such as those to improve its photo searching features. As great as the technology is, Facebook is seeking something better: the ability to pull text from images and understand the text.

Like this: