SNEAKER 0, REIMAGINING THE MUSEUM EXPERIENCE WITH MACHINE LEARNING AND AUGMENTED

TitleSNEAKER 0, REIMAGINING THE MUSEUM EXPERIENCE WITH MACHINE LEARNING AND AUGMENTED
BrandTHE DESIGN MUSEUM (THOUGH SNAP INC.)
Product/ServiceSNEAKER UNBOXED: STUDIO TO STREET EXHIBITION
Category A01. Innovative Technology
Entrant GOSPOOKY Amsterdam, THE NETHERLANDS
Idea Creation SNAP London, UNITED KINGDOM
Idea Creation 2 GOSPOOKY Amsterdam, THE NETHERLANDS
Production GOSPOOKY Amsterdam, THE NETHERLANDS
Post Production GOSPOOKY Amsterdam, THE NETHERLANDS
Credits
Name Company Position
Tom Geurts GoSpooky Tom Geurts was Technical Director of the development for Sneaker 0. In this role he managed and oversaw progress and outcomes of the machine learning, design and AR team and was responsible in bringin

Why is this work relevant for Innovation?

Introducing Sneaker 0 - the world’s only sneaker to be designed by a computer and launched solely in AR on Snapchat. Sneaker 0 is the result of a partnership between Snap and social-first marketing company GoSpooky. Joining forces to reimagine the Design Museum’s ‘Sneakers Unboxed: From Studio to Street’, with the aim to realise an innovative museum experience that would draw in new crowds and would create media buzz. Sneaker 0 was brought to life by leveraging Snap’s innovative camera tools, and combining this with various layers of technology, from Machine Learning, to Augmented Reality and 3D design.

Background

With the onset of a global pandemic, museums and galleries worldwide had to close their doors. As the world started to cautiously open up again, the Design Museum in London was looking for an innovative experience that would draw in crowds and create media buzz in the UK and across borders. The Design Museum reached out to Snap Inc. looking for support in realising these objectives through the Snapchat platform. Consequently Snap and GoSpooky joined forces to reimagine the museum experience for the Design Museum’s Sneakers Unboxed: From Studio to Street. GoSpooky and Snap received carte blanche from the Design Museum. Through the use of Machine Learning, Augmented Reality and high-end 3D design Sneaker 0 was born, and brought to life on the Snapchat platform.

Describe the idea

Due to the onset of the global pandemic, museums and galleries worldwide had to close their doors. As the world started to cautiously reopen, the Design Museum in London was looking for an innovative experience that would draw in crowds and create media buzz around its venue. And so, a partnership between Snap and GoSpooky was born, to reimagine the Design Museum’s ‘Sneakers Unboxed: From Studio to Street’. The exhibition featured 300 of the most iconic sneakers, so we thought, what if sneaker 301 were to be featured at the exhibition: a whole new iconic sneaker created entirely by a computer and launched using AR? To bring Sneaker 0 to life, we brought together several layers of technology and innovation, from Machine Learning (ML) and Augmented Reality (AR) to 3D design, giving audiences something they’d never seen before and a reason to get back out and involved in cultural activities.

What were the key dates in the development process?

Strategy (19-03-2021 - 26-03-2021) ‘Sneakers Unboxed’ was a curated exhibition featuring 300 of the most iconic sneaker designs ever created. So we thought, what if a 301st sneaker were to be featured at the exhibition: a whole new iconic sneaker created entirely by a computer and launched using AR? With AR being one of the most popular features of the Snapchat app which reaches 90% of Gen Z in the UK everyday, it was a clear winner for the most impactful tool to reach new crowds. That, and the fact that the app’s second largest user base is Millennials up to 35 years old. Both are large demographics hungry for experiences that seamlessly link the physical and digital through the power of the camera. It’s camera-centric platforms such as Snapchat that are the driving force of AR’s development. Building ML models which can recognise body parts such as feet, for example, are efforts helping to enhance the way people interact with the world around them. AR and ML are reshaping consumer experiences to inform, inspire and excite on another level. These insights formed the concept for a 301st sneaker for the exhibition that would be designed using ML, built by a computer and launched in AR on Snapchat. But how do you train a computer in sneaker design? GoSpooky led the process as outlined below: 1. Data collection (09-04-2021 - 21-04-2021) Thousands of different images of sneakers were collected that spanned an entire sneaker history. Over 233.428 images to be exact. These were resized allowing the computer to process them; otherwise known as data preprocessing. 2. Neural network (22-03-2021 - 08-04-2021) But before the computer could learn, it needed to know how to learn. A neural network was used to train the computer to learn how to interpret the images it would be shown. The output is a machine learning model which is able to identify patterns and trends in the images. For example, sneaker silhouettes, colourways, and fabric structure. The computer was trained using a cutting-edge generative model* that can both assess and create images. Meaning the more pictures of sneakers it would be shown, the better it would get at generating new images of unique sneakers. 3. Sneaker education (08-04-2021 - 18-04-2021) Next, it was time for the machine learning model to learn. By showing it the thousands of images of sneakers, the model was trained to understand what a sneaker looks like. 4. Sneakerhead machine (18-04-2021 - 28-04-2021) Then, one by one, the model was fed images of the 300 iconic sneakers in the Sneakers Unboxed exhibition to fine-tune its knowledge of great sneaker design. It was given time to process each shoe, so it could recognise the importance of the iconic sneakers and their characteristics. 5. The birth of Sneaker 0 (29-04-2021) Finally, it was ready to create its own sneaker. But the first reveal wasn’t exactly what we expected. As half of the sneakers in the exhibition were second-hand, it meant that the machine had taught itself to design second-hand sneakers. After giving it a little bit more time to learn, the machine delivered its final design. Enter the birth of the ultimate sneaker: Sneaker 0. Named because no humans were involved in its design. The first sneaker designed entirely by a computer and an amalgamation of the best of sneaker history. How? Because the machine learning model had been trained to understand which place each of the 300 iconic sneakers occupied within its realm of sneaker knowledge.** So every single iconic sneaker played an equal part in Sneaker 0’s design. 6. Exploring the creation (30-04-201 - 03-05-2021) A second machine learning model*** was used to explore the ‘mind’ of the computer. This provided access to its understanding of colour, shape, and rotation, informing GoSpooky and Snap of how Sneaker 0 looked from various angles and in different colourways. In other words, it was a way to pick the designer’s brain. 7. Launching Sneaker 0 in the ecosystem (21-05-2021 - now) The next step was to bring Sneaker 0 to life in an ecosystem that would resonate with the right audiences - those the Design Museum was aiming to reach for Sneakers Unboxed as well as future exhibitions. A dynamic Snapchat lens would allow users to virtually try on Sneaker 0 for size in AR, deepening engagement with the design and stimulating shareability. It also allowed visitors to take a piece of the exhibition home, adding a new dimension to the museum experience and making it accessible to a wider audience. Sneaker 0 could be experienced by simply opening Snapchat and scanning a Snapcode. Once the AR Lens has been unlocked, the user could tap on the screen until the shoe would form. Then they could point their smartphone camera at their feet (or a friend’s) and Sneaker 0 would appear, fitting perfectly. Snapchatters could then browse through the various colourways to try them all. While the shape, materials and textures of Sneaker 0 remained the same, it was available in four colourways to encourage Snapchatters to experiment. The first, the ‘OG’, was a mixture of white and grey. To provide an even more immersive experience, a ‘Dynamic’ colourway was integrated in three variations. Adaptive, it adjusted its colour in real-time based on the shoes of the wearer, and the user could tap to select how much colour to add to the design. 8. Putting Sneaker 0 on the map (02-08-2021 - now) Finally, Sneaker 0 was also quite literally put on the map - Snap Map, that is. By searching for the Design Museum users are able to try on Sneaker 0 for size, wherever they are. Bringing Sneaker 0 and the Design Museum to the world through immersive technology and a forward-thinking social platform. *The generative model is Nvidia’s StyleGAN2-ada. GANs, or generative adversarial networks, are trained to generate images using two network models. One is the ‘discriminator,’ which determines whether an image is real or generated, and the other is the ‘generator,’ which generates new images. **This process of plotting the iconic sneaker data within the model’s sneaker knowledge space is called reprojection ***SeFa - Closed-Form Factorisation of Latent Semantics in GANs

Describe the innovation / technology

To build Sneaker 0, a computer was trained in sneaker design by scanning thousands of sneakers. Then, it was given visual information of the 300+ iconic sneakers displayed at the exhibition, to learn what makes a great design. This information was fed into a machine learning model, which led to the computer creating its version of the ultimate sneaker. Sneaker 0, the world’s only sneaker to be designed using Machine Learning, built by a computer and launched solely in Augmented Reality on Snapchat. Using Snap’s AR technology, users can virtually try on Sneaker 0 by opening Snapchat and scanning the Snapcode. Once the AR Lens has been unlocked, sneakerheads tap on the screen and once the shoe forms, they can then point their smartphone camera at their feet and Sneaker 0 will appear, fitting perfectly. Snapchatters can cycle through various colour options to try them on and experiment: The OG - a mixture of white and grey elements chosen by the computer Dynamic I, II and III - an adaptive sneaker that adjusts its colour in real time based on the shoes - you’re wearing - while the user can tap to select how much colour to add the design.

Describe the expectations / outcome

Sneakers Unboxed and Sneaker 0 - the world’s only sneaker to be designed by a computer and launched solely in AR on Snapchat - successfully created a moment for the Design Museum that traditional media could get behind, whilst also placing the story in the hands of new audiences. Leveraging Snap’s innovative camera tools that push the boundaries of tech and creativity, the traditional museum experience was transformed into an immersive one that created real impact. The Design Museum recorded its second-highest exhibition presale ever and completely sold out all its weekend tickets - in spite of a cautious nation and visitor number restrictions.