About Native Instruments
For over 25 years, Native Instruments has been at the forefront of musical innovation with technology that has revolutionized how people make music. They create audio content, virtual instruments, software plug-ins as well as hardware to millions of creators worldwide, from award-winning artists to first-time producers.
Native Instruments’ Challenge
Andy Sarroff, Senior Research Manager at Native Instruments, leads a research team that creates virtual instruments and synthesizers, in addition to software for audio production. Native Instruments sells both the software-based instruments that make sounds, as well as access to their expansive catalog–or rather, digital library of sounds.
Sarroff and his team have worked for some time to improve the discoverability of audio files within the library, knowing that end-user creators searched in a multitude of ways, using varied descriptors. One critical piece that Sarroff and his team hadn’t yet accounted for was how to provide social context insights in the search and discovery process.
“We knew that social context was going to be a huge part of delivering a smarter, more intuitive search. Of course, GenAI was the likely solution, but that wasn’t a space we specialized in,” said Sarroff.
Because Sarroff and his team primarily worked with Machine Learning (ML) and digital signal processing and already had a backlog of business-critical projects in motion, they knew they’d need a third party expert to develop and fast track a GenAI strategy for addressing this challenge.
Why Tribe AI?
“Of course, there are plenty of other consultancies specializing in GenAI and LLMs. It was especially important to us, however, that we partner with people who share the empathy we have for users in the music space,” said Sarroff.
Native Instruments discovered Tribe AI through a recommendation from their trusted tech investor Francisco Partners. Francisco Partners knew from experience that Tribe AI team members were not just experts on the technology. The Tribe AI team regularly demonstrated their commitment to understanding the end user’s challenge as a means to developing truly impactful solutions.
Proposed Solution
Native Instruments and Tribe AI began a 4-week proof of concept (POC) engagement centered on building a prototype of a human-like chat interface that would deliver an exceptional user search and discovery experience specifically focused on improved recognition of social context.
The GenAI strategy would rely on Amazon Bedrock + Native Instruments’ proprietary indexed sound file data; and would utilize Large Language Models (LLMs) to provide improved search results to end user creators. This technology would allow creators to use more human-like language when searching the library. It also would recognize that a user search for “Arcade Fire” should produce sounds similar to the style of the band named “Arcade Fire,” rather than those that sound like an “arcade” or a “fire”. Sarroff believes the chat interface could really revolutionize the creator experience, as it's a GenAI application never before implemented in this space.
“AI is a very touchy subject for the creative workforce because there are some companies trying to replace creative talent. We are invested in serving the creators, not disenfranchising them. We want to empower them to realize their creative outcomes as quickly as possible,” said Sarroff.
Tech Stack Details
Full-stack cloud-based application working alongside the existing Native Instrument environment:
- Cloud: Amazon Web Services (AWS) Bedrock
- Large language models: Anthropic Claude 2
- Audio <> Text Embedding models: Contrastive Language-Audio Pretraining (CLAP)
- Programming languages: Python + Typescript
How It Works
The project backend is in Django, and the front-end is in React JS. Specifically, the backend was built via a GitHub action, containerized using Docker and pushed to Amazon Web Services (AWS) Elastic Container Registry (ECR). An EC2 instance was created which will run the docker container. In the future, this will instead be auto-scaled by using AWS Elastic Container Service (ECS).
For DNS routing, the teams used Route53, and for security, they utilized AWS Elastic Load Balancing (ELB) which defines the firewall rules.
The backend interacts with AWS Bedrock, to perform LLM queries using Claude. The LLM transforms queries into suitable text for searching via a CLAP model (Contrastive Language-Audio Pretraining). The CLAP model currently lives outside of the AWS infrastructure, with plans to move that into SageMaker.
The frontend is deployed via a GitHub action, which packages the NPM build and pushes it to an S3 Bucket. Additionally, the teams are using Route53 for DNS routing.
Developing the Roadmap
The engagement spanned four weeks, with the first two weeks focused primarily on discovery and scope development.
The Tribe AI team learned that the Native Instruments digital library was quite extensive and the audio files within it had been tagged with indexable data. These descriptor tags could certainly be leveraged, but on their own wouldn’t deliver the smarter search and discovery interface that Native Instruments desired. The Tribe AI team planned to couple the indexed data tags with GenAI to produce a more human-like interface that would be able to understand more nuanced prompts and pop culture references.
The interface would be powered by Anthropic and would rely on LLMs to deliver the additional social context layer that had been missing from the previous solution. Sarroff and his team expressed relief in the idea of leaning on the GenAI strategy–and appropriate tech partners–rather than having to create the social context connection layer themselves.
“Working with Tribe AI on this project has brought so much value. This kind of work sits outside of our circle of experience. Without having to completely divert our team’s attention, we were able to develop the prototype and learn a lot via osmosis,” said Sarroff.
The final two weeks of the POC engagement involved rapid prototyping for the Tribe AI team. Sarroff knew he’d have some work to do before the testing phase in order to ready his internal team. Sarroff made plans to present on the capabilities of the prototype along with details on the POC engagement work and his team’s learnings from the process. He hoped the presentations would reconfirm the team’s trust in the GenAI strategy and bolster idea generation for developing monetization models for the new interface.
Tribe Team Members
Arushi - Product Manager
Marius - GenAI Lead
Craig - Engagement Lead
Native Instrument’s Experience Working with Tribe
Sarroff and his team had long before identified the technical gap in their current search interface. They also knew that GenAI and the use of LLMs were likely the source of the solution. Where the Tribe AI team brought tremendous value was in narrowing down the problem space enough to provide expert guidance on the technology that resulted in a working prototype of the human-like chat interface. The prototype will allow Sarroff and his team the ability to perform A/B testing against their existing solution.
“Thanks to the Tribe AI team, we were able to quickly ideate and test new AI concepts for the benefit of our creative audience of music producers and composers. The Tribe AI team demonstrated deep expertise in AI and a commitment to rapid execution,” said Sarroff.
Impact
The POC engagement was successful in developing a functional human-like chat interface that better addresses the social context shortcomings of the existing search and discovery process. Leveraging an Amazon Bedrock-powered GenAI + LLM deployment strategy allows the end users to attain nuanced or musically-inclined responses. Previously, it wasn’t uncommon for users to have to make multiple attempts at a search in order to gain their desired response.
Based on the outcomes of the POC engagement, Sarroff affirmed that focusing on this deliverable was the right choice for exploring GenAI-centered innovation for Native Instruments. He has shared initial results and prototype details internally at Native Instruments in presentation format, and plans for an upcoming A/B testing phase where leadership can interact with the interface. Sarroff believes his leadership team will have a critical eye and valuable feedback to share as they test the prototype.
The Future
“We are living and breathing AI right now. It exists all around us and it's not going away. It's part of our new life stack,” said Sarroff.
Sarroff and his team look with optimism at the ways they predict GenAI can further innovate tools for the music industry. Leaning on the same technology that improved the search and discovery interface for creators, Sarroff sees revolutionized workflows for amateur music producers as they more innately discover production techniques and products to further streamline the overall creative process.
“The technology is versatile. We can see it being used almost anywhere: from product catalogs and the software, to the ecommerce website, or even in the instruments themselves. Every product we are working on would be less powerful without this piece,” said Sarroff.
