Voice: For interactive voice interfaces, how do you bring to market a smart-speaker skill that generates a high CSAT score from Day 1?
A niche team inside a larger company had already captured ~40MAU. The product is very specialized but has broad use cases. This breadth was seen as an opportunity and several companies are expanding into the market with “add on” products.
Engineers wanted to launch a smart-speaker skill but product management was feeling the pressure to fight back the lazy competition.
THE ASSIGNMENT
Go to market with a differentiated voice experience that generates CSAT >= 79%
THE STRATEGY
Engineers had already designed the algorithm for constructing smart-speaker responses. Product was tasked with generating the various parts of the smart responses (i.e., the content).
Since the actual skill (product) was still being coded, having people test a beta was not possible. In other words, we couldn’t have participants engage (“<smart speaker>, <what ___>?” or “<smart speaker>, <do some thing>.”) and get feedback on the experience.
To test the content, we reversed the typical smart speaker process. We did not ask participants to rate a response after asking or telling the smart speaker something. Instead, we gave the response first, “Imagine your smart speaker just told you this: <mp3 file plays a verbal response>”. Then we asked the participants, “What do you think you said to your smart speaker?“. This study was duplicated more than 50 times with iterations on previous responses and creation of new responses.
This reversal gave the team a breadth of smart speaker commands from which people would expect to hear the response that the team had crafted for the skill.
THE RESULTS
- In about a dozen cases, completely unexpected commands were discovered that would have triggered an error. New responses were crafted to avoid the error response. Currently, the only smart speaker skill that does not play an error for these commands.
- We’ve been told that the CSAT scores are better than expected.
- Engineering and product management experienced unprecedented alignment.