Google lets you experience what it’s like to have synesthesia

Imagine hearing images

What you need to know

• The Centre Pompidou and Google Arts & Culture have teamed up to help visitors learn about synesthesia.
• The partnership has resulted in the development of a machine learning experiment that lets anyone “hear” a Vassily Kandinsky painting.
• To achieve Kandinsky’s sound, the company partnered with music artists Antoine Bertin and NSDOS.

Google Arts & Culture and the Centre Pompidou in Paris, France have teamed up to offer visitors a chance to learn more about the life, art, and unique perceptual abilities of Vassily Kandinsky. Kandinsky is often credited with being one of the pioneers of abstract art, and he is also acknowledged as a person who had a special multi-sensory ability called synesthesia. Synesthetes, or those with synesthesia, profess the ability to hear colors and shapes and translate music and sounds to visual media. Lest you think it an obscure condition, even modern artists like Billie Eilish and Pharrell Willams are also said to be synesthetes.

The new Sounds like Kandinsky project was created as an attempt to explain this phenomenon and allow virtual visitors to experience his art as close as possible to the way he once did. Contemporary music artists Antoine Bertin and NSDOS worked with Google Arts & Culture and Centre Pompidou on the project.

We trained Google’s Transformer neural network, which was developed by the Google Magenta team, on music from Kandinsky’s time and invited this machine to generate new scores to form a new perspective on what the painting may sound like from today’s point of view. — Antoine Bertin

In addition to background on Kandinsky’s life and work, this virtual exhibit lets you hear what Kandinsky might have heard when he created and viewed his masterpieces. You can zoom in and click on various parts of his Yellow Red Blue (1925) painting to get a sense of what different colors and textures sounded like in his head.

This isn’t the first interactive exhibit for Google Arts & Culture. Over the Christmas holidays, Google allowed users to experience the funny Blob Opera, and the company continues to add new AR interactive creatures in its Arts & Culture app. Of course, one of our favorite features is the ability to add famous pieces of art as wallpaper on the best Chromebooks and best Android phones.