Mike Schroepfer, CTO at Facebook announced that the social media giant is working on a whole new approach to how users can search for subject of interests. Facebook will understand photos and will be able to tell you quite a story  about them.

You are now wondering what does this mean. Facebook will understand photos? What is there to understand? I just uploaded of photo of me doing what I like and that’s that. Well, in reality, things are a bit different. In order to facilitate the user’s access to certain content, Facebook will employ the use of an advanced AI concept capable of rendering images into words.

In order to demonstrate how the software actually works, representatives from Facebook provided us with an example. In front of us we have the picture of a guy trying to do a rail grind with his skateboard. The description seems to be quite elliptical (the guy wrote “dudeeeeeeeeeeee!”). So how can Facebook help to better understand what’s going on in the picture?

Basically by using syntactical elements, Facebook breaks down the pictures into elements which can, in term, help us to decipher the picture. Each picture is broken down into basic syntactic compunds like noun phrases, verb phrases and prepositional phrases. Now, in the noun phrase category, the algorithm enumerates all nouns found in the picture. In our case, under the NP category we can find nouns such as “skateboard”, “a man”, “trick” or “his skateboard.” Basically, the engine identifies and names everything he can find in the picture.

The Verbal Phrase section is filled with action verbs, or in our case, verbs attributed to moving, that can define the action of riding on a skateboard. Under VP we can see stuff like “doing”, riding”, “performing” or flying”.

As for the Prepositional Phrase section, this section delivers additional information to the viewer concerning why and when the action is taking place. The PP section reads out prepositions like “on”, “of”,”in”,”at” and “with”.

After the system has finished the process of breaking down your picture into words, he reconstitutes the whole context in a phrase or a sentence. For the example provided by Google, the end sentence will be something like: “A man riding a skateboard on ramp.”

Facebook has issued no statement regarding when the update will be available. But the prospect of having your own AI capable of breaking down pictures into words is simply astounding.

Image source:freepik.com