With every company worth its salt boasting of at least one or two in its kitty, there is certainly no dearth of voice models in the market. While many of these are impressive, the common thread that binds them all is their struggle to achieve seamless and emotionally resonant interactions. But, a recently released end-to-end, full duplex voice call model from the CEO of Soul, Zhang Lu, and her team is being positioned as the solution for these shortcomings.
The Origins of Soul App’s New Voice Model
The roots of this new voice model can be traced to the social platform’s long-standing commitment to using groundbreaking technologies. From the get-go, the core team has been working on developing in-house technologies to ensure that the users of Soul Zhang Lu’s platform are offered emotionally valuable experiences when using the app.
The Role of AI in Soul App’s Growth
Soul was made available to its users in 2016 and since then the company has consistently used artificial intelligence to power most of its popular features. In the beginning, AI was used to connect users with like-minded folks who shared their personalities, passions, and interests. Over time, AI started playing a more central role as Soul Zhang Lu’s engineers independently developed large language and voice models.
Emotional Intelligence as a Core Objective
Soul App’s vice president, Che Bin, recently stated that the core goal of the platform’s team is to create and maintain a product market fit. He further explained that to do so, the company is working in earnest on creating AI models that have just as much emotional intelligence as general intelligence. According to Che Bin, models that have a high emotional quotient are the only way forward, particularly in terms of their integration into social scenarios.
Using this goal as the cornerstone for independently developed models, the engineers of Soul Zhang Lu submitted a one-of-its-kind competition entry at MER 24. Their submission was in the SEMI category, which dealt with unsupervised learning techniques. What made the submission stand out was that the team had devised an ingenious way to use unlabeled data by way of pseudo labels. In effect, this remedied the problem of finding labeled data that is needed to train models for emotional intelligence but is both in short supply and extremely expensive.
Handling Competing Data Inputs from Various Modalities
Another aspect that the submission successfully handled, which is also a prime issue with training AI models, was that of competing data inputs from various modalities. The win garnered praise for Soul on the international stage, and this new voice model has attracted the attention of industry watchers.
Full Duplex Capabilities: Mimicking Human Comprehension
The end-to-end voice model boasts full duplex capabilities, which means that the algorithm can recognize simultaneous speech coming from the user and the machine. This is exactly how human comprehension works. The ability to understand its responses and the user’s reactions to it allows the machine to respond and even change the track/manner/subject of interaction at rapid speed. This gives the interaction a marked human-communication-like quality.
Generating Human-Like Responses with No Cascaded Processing
Furthermore, the model is capable of generating and responding to the minutiae of regular interactions that lend them a human quality, such as response tokens, discourse markers, and even interruptions. Additionally, the model developed by Soul Zhang Lu’s team does not involve cascaded processing methods that introduce inefficiencies and limit the natural flow of dialogue.
End-to-End Modeling: A Disruptive Upgrade
Instead, it is designed for end-to-end modeling from speech to speech, which is undoubtedly a disruptive upgrade to the speech interaction system. This new model eliminates the need to flow through multiple stages such as “speech recognition, natural language understanding, and speech generation,” allowing for direct speech input-speech output. Thus, end-to-end modeling maximizes information transmission without loss and reduces response latency.
Real-Time Communication with Emotional Depth
So unlike traditional voice models that are prone to noticeable delays, robotic or overly scripted responses, and a lack of emotional understanding that leaves users feeling disconnected from the conversation, Soul Zhang Lu’s model offers a real-time communication experience with emotional depth that feels closer to speaking with an actual person.
Personalized and Empathetic Responses
Apart from ultra-low latency, this model has an acute understanding of emotional cues and is able to respond to them in a life-like manner. This makes the interaction more personalized and empathetic. Plus, Soul Zhang Lu’s team has managed to give the model the ability to render voices in an ultra-realistic manner, enhancing the immersive quality of the interactions.
Reducing Information Loss and Enhancing Accuracy
Because this is an end-to-end model, there is a lower loss of information as well as a reduced potential for errors. Also, the lack of intermediate processing stages equates to improved accuracy in interactions and a seamless flow of conversation.
Versatility in Real-Life Interactive Scenarios
If all of that wasn’t impressive enough, Soul’s model can understand user speech even in complex environments and with diverse accents. Moreover, this model from Soul Zhang Lu’s team can simulate animal sounds from the physical world, understand multi-person conversations, and achieve multi-style language switching, literary content creation, and impromptu singing. In essence, it is very close to handling most of the needs of real-life interactive scenarios.
Testing Phase and Platform Integration
At this time, the model is being beta tested, and through this stage, when its performance was measured for latency, it fared better than the industry average. Currently, voice models are being used by Soul’s team to power several of the platform’s well-loved features, such as the in-app assistant and chatbot AI Goudan, as well as the AI players of Werewolf Awakening.
Elevating User Experience to a New Level
While these features are already capable of humanistic responses, this new model will, without a doubt, elevate the experience of the users to a whole new level. Soul users who had participated in a survey conducted by the platform earlier in the year had expressed the desire to befriend AI characters. This new voice model will likely make it possible for the team of Soul Zhang Lu to take several significant steps in the direction of fulfilling this user demand.
