Voice Search Optimization

Voice is the next frontier for AI Agents, but most builders struggle to navigate this rapidly evolving ecosystem. After seeing the challenges firsthand, I've created a comprehensive guide to building voice agents in 2024. Three key developments are accelerating this revolution: -> Speech-native models - OpenAI's 60% price cut on their Realtime API last week and Google's Gemini 2.0 Realtime release mark a shift from clunky cascading architectures to fluid, natural interactions -> Reduced complexity - small teams are now building specialized voice agents reaching substantial ARR - from restaurant order-taking to sales qualification -> Mature infrastructure - new developer platforms handle the hard parts (latency, error handling, conversation management), letting builders focus on unique experiences For the first time, we have god-like AI systems that truly converse like humans. For builders, this moment is huge. Unlike web or mobile development, voice AI is still being defined—offering fertile ground for those who understand both the technical stack and real-world use cases. With voice agents that can be interrupted and can handle emotional context, we’re leaving behind the era of rule-based, rigid experiences and ushering in a future where AI feels truly conversational. This toolkit breaks down: -> Foundation layers (speech-to-text, text-to-speech) -> Voice AI middleware (speech-to-speech models, agent frameworks) -> End-to-end platforms -> Evaluation tools and best practices Plus, a detailed framework for choosing between full-stack platforms vs. custom builds based on your latency, cost, and control requirements. Post with the full list of packages and tools as well as my framework for choosing your voice agent architecture https://lnkd.in/g9ebbfX3 Also available as a NotebookLM-powered podcast episode. Go build. P.S. I plan to publish concrete guides so follow here and subscribe to my newsletter.

Cartesia Sonic-3 is the first AI voice model I’ve seen that nails Hindi perfectly. For years, even the best text-to-speech (TTS) models struggled with Hindi. The rhythm, tonality, and emotional micro-expressions just didn’t sound human and the accent was inaccurate. This model doesn’t just translate Hindi. It is specially trained for it, with precise control over pacing, expressions and  tonality, all rendered in real time. Under the hood, Sonic-3 is engineered for low-latency voice generation optimized for conversational AI agents, clocking in 3–5x faster than OpenAI’s TTS while maintaining superior transcript fidelity. What makes it stand out technically: → 𝗚𝗿𝗮𝗻𝘂𝗹𝗮𝗿 𝗰𝗼𝗻𝘁𝗿𝗼𝗹 𝘁𝗮𝗴𝘀 let developers dynamically modulate speed, volume, and emotion inside the transcript itself. ("Can you repeat that slower?" now works in production.) → 𝟰𝟮-𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗺𝘂𝗹𝘁𝗶𝗹𝗶𝗻𝗴𝘂𝗮𝗹 𝗺𝗼𝗱𝗲𝗹 built on a single unified speaker embedding, so one voice can switch between languages like Hindi, Tamil, and English natively while maintaining accent continuity. → 𝟯-𝘀𝗲𝗰𝗼𝗻𝗱 𝘃𝗼𝗶𝗰𝗲 𝗰𝗹𝗼𝗻𝗶𝗻𝗴 powered by a low-sample adaptive cloning pipeline that enables instant personalization at scale. → 𝗥𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝘀𝘁𝗮𝗰𝗸 achieving sub-300 ms end-to-end latency at p90, tuned for live interactions like support agents, NPCs, and healthcare assistants. → 𝗙𝗶𝗻𝗲-𝗴𝗿𝗮𝗶𝗻𝗲𝗱 𝘁𝗿𝗮𝗻𝘀𝗰𝗿𝗶𝗽𝘁 𝗮𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁 that handles heteronyms, acronyms, and structured text (emails, IDs, phone numbers) which usually break realism in production systems. 🎧 Here is example of me trying Sonic-3’s Hindi. You have to hear it to believe it. If you’re building voice agents, conversational AI, or multimodal assistants, keep an eye on Cartesia. They’ve raised $100M to build the most human-sounding voice models in the world, and Sonic-3 just set a new benchmark for multilingual voice AI. #CartesiaPartner

Building AI is easy. Running it when OpenAI goes down is the real test. That line perfectly framed a tech meetup at Meesho, Bangalore — where the conversation wasn’t about demos, but about operating AI in the real world. Three talks. One clear theme: production reality beats lab intelligence. 1) AI at Billion Scale – Portkey | Ayush Garg (Co-founder) At scale, AI stops being a model problem and becomes a systems problem. Key lessons: - Resilience > intelligence - Stability > sophistication - Abstraction layers are unavoidable at scale - Security belongs at the gateway, not in apps Hard truths: - Blind retries = 2x cost, 3x latency - FINOPS asks: Which team consumed these tokens? - CISOs ask: Can you audit every request? - Business asks: Why did AI costs spike 4x? Observability. Accountability. Audit trails. Guardrails. At billion #scale, AI is infra first, models second. 2) Agents for #Agents – Anubhav Singh (AI Engineer @ Weights & Biases) Agents are powerful. Agents without evals are dangerous. What mattered: Agent evals using role assignment + tools LLM-based tracing with Weave from Weights & Biases. Metrics that actually reflect reality: • Tokens consumed • TTFT + TPOT • Call-level traces Standout demo: An optimizer agent improving response quality. Polite refusal trait improved from 20% → 60%. Not prompt engineering. Measured, observable improvement. 3) #Voice AI at Scale – Anuj Goel (Tech Leader at Meesho & VoiceBot) Voice AI is where all constraints collide: latency, language, and real users. The stack: - Telephony layer - VoiceBot pipeline: ASR → VCA (intent detection) → TTS - Fine-tuned LLMs to bridge human–agent conversation gaps - Models are a mix of internal and external nature Scale & performance: - 2 lakh calls/day on average, peaks at 5–10 lakh - p90 latency ~1 second - 3 Indic languages supported (Tamil, Telugu, Malayalam) + English & Hindi Key engineering choices: - Cross-region fallbacks & dynamic routing - Circuit breakers & vendor localisation - Real-time escalation using parameters like empathy & coherence - LLM streaming architecture to cut latency - Warmed-up WebSockets for faster starts - Metadata (order IDs, details) cached deliberately - Conversation phrasing intentionally not cached to mimic the natural flow of human agent behavior This wasn’t a chatbot story. This was distributed systems engineering with humans in the loop. Big takeaway from the morning: If you can’t trace it, audit it, explain its latency, and justify its cost — it’s not production AI. AI isn’t scaling because models got smarter. It’s scaling because engineering discipline finally caught up. Respect to Meesho and Nihal Kashinath & #DeepTechStars for hosting AI conversations that actually matter. Thank you for having me! #AbhiWritesAI #AIatScale

VoiceTextBlender introduces a novel approach to augmenting LLMs with speech capabilities through single-stage joint speech-text supervised fine-tuning. The researchers from Carnegie Mellon and NVIDIA have developed a more efficient way to create models that can handle both speech and text without compromising performance in either modality. The team's 3B parameter model demonstrates superior performance compared to previous 7B and 13B SpeechLMs across various speech benchmarks whilst preserving the original text-only capabilities—addressing the critical challenge of catastrophic forgetting that has plagued earlier attempts. Their technical approach employs LoRA adaptation of the LLM backbone, combining text-only SFT data with three distinct types of speech-related data: multilingual ASR/AST, speech-based question answering, and an innovative mixed-modal interleaving dataset created by applying TTS to randomly selected sentences from text SFT data. What's particularly impressive is the model's emergent ability to handle multi-turn, mixed-modal conversations despite being trained only on single-turn speech interactions. The system can process user input in pure speech, pure text, or any combination, showing impressive generalisation to unseen prompts and tasks. The researchers have committed to publicly releasing their data generation scripts, training code, and pre-trained model weights, which should significantly advance research in this rapidly evolving field of speech language models. Paper: https://lnkd.in/dutRcaAA Authors: Yifan Peng, Krishna C. Puvvada, Zhehuai Chen, Piotr Zelasko, He Huang, Kunal Dhawan, Ke Hu, Shinji Watanabe, Jagadeesh Balam, Boris Ginsburg #SpeechLM #MultimodalAI #SpeechAI

We now know what Thinking Machines has been up to, they just shared a research demo of their first model, TML-interaction-small! This is likely the first voice models you could reliably play a game of charades with. The model is designed to allow two way continuous communication by having a continuous stream of micro turns of 200ms to allow responses to text, images and videos. Current multimodal voice models struggle with latency and accuracy of a continuous stream of information, usually requiring a custom harness to prompt the models when to respond. There aren't too many benchmarks that can capture the interactions, and the team writes about a few they've built internally (quoted): TimeSpeak: Tests whether the model can initiate speech at user-specified times while producing the correct content. For example: “I want to practice my breathing, remind me to breathe in and out every 4 seconds until I ask you to stop.” CueSpeak: Tests whether the model speaks at the appropriate moment with the expected semantically correct response. Dataset entries are created to ensure that the model needs to speak at the same time as the user to get a full score. For example: “Everytime I codeswitch and use another language, give me the correct word in the original language.” In terms of the architecture,TML-Interaction-Small is a 276B parameter transformer MoE with 12B active parameters that was pre-trained from scratch without using an encoder model for audio and video components. The decoder then generates the 200ms excerpts in a text and audio format. Performance over a long context still remains a challenge and team has deferred running larger versions of the model due to significant interaction tradeoffs. It was also interesting to see how some of the earlier shared work like batch invariance was used to improve training stability. Definitely recommend watching the demos and reading the blog post, they're quite neat! Looking forward to the release later in the year https://lnkd.in/ehDDN7uh

Voice search optimization is becoming increasingly important as search trends evolve. With the rise of virtual assistants like Siri, Alexa, and Google Assistant, users are now more likely to use voice commands to search for information online. This shift means businesses need to adapt their SEO strategies to cater to this changing landscape. Voice search differs from traditional text-based search in several ways. It tends to be more conversational and natural, often resulting in longer queries with specific questions. As a result, businesses need to optimize their content to match these conversational queries and provide direct, concise answers. To effectively optimize for voice search, businesses should focus on creating content that answers common questions related to their products or services. This includes using natural language and long-tail keywords that reflect how people speak in everyday conversations. Additionally, ensuring that your website is mobile-friendly and loads quickly is essential, as many voice searches are conducted on mobile devices. By adapting to changing search trends and optimizing for voice search, businesses can improve their visibility in search engine results and better meet the needs of their target audience.

Most voice AI is just a chatbot with a microphone. One company was purpose-built for real phone calls from day one. The architecture lesson most AI builders learn too late: Everyone builds text agents first. Winners build voice first. Text agents get retries. Formatting. Autocorrect. Voice agents get one shot. Real-time. No edits. Then production hits ↓ → Latency: Model takes 3 seconds. Customer hangs up. → Context: "Uh, yeah, so I need to, wait — can you also check my..." → Interruptions: Humans talk over each other. Chat agents break. → Compliance: Every voice interaction is regulated differently. Two traps I see teams fall into: Trap 1: Bolt STT onto a chat agent. Add TTS on output. Call it "voice AI." That's a wrapper. Wrappers break in production. Trap 2: Build your own with Pipecat, LiveKit, Vapi. 6 months later you're managing STT providers, TTS rate limits, LLM deprecations, infrastructure scaling, compliance audits. You wanted a voice assistant. Now you're a voice infrastructure company. PolyAI solved this differently. Full stack built for voice since 2017: → Proprietary ASR + LLM trained on real customer service transactions → 45+ languages. 24/7. Unlimited scale. → Handles surges instantly — storms, outages, promos — zero staffing panic Not just handling calls — generating revenue: → Turning bookings into room upgrades → Enrolling callers into rewards mid-conversation → QA Agents scoring every call automatically → Analyst Agents surfacing patterns no human team catches One healthcare company found fewer complaints from the AI than human reps — on the hardest, most emotional calls. Marriott. FedEx. Caesars. PG&E. 25+ countries. 391% ROI. $10.3M average savings. Payback under 6 months. The companies still running "press 1 for sales, press 2 for support"? Not behind on technology. Behind on architecture. That gap compounds every quarter. My stress test for any voice AI: → Noisy environment → Regional accent → Language switch mid-sentence → Multi-step transaction → Worth 15 minutes if you're evaluating: https://poly.ai/gordon Build or buy — what's your current approach to voice?

Tencent just open-sourced a 7B model that listens and talks at the same time. No ASR. No TTS. One model does everything. ━━━━━━━━━━━━━━━━━━━ Covo-Audio. Full-duplex audio LLM. Interrupt it mid-sentence — it responds naturally. Like a real conversation, not turn-based. → Qwen2.5-7B backbone + Whisper encoder → End-to-end — audio in, audio out, no separate ASR→LLM→TTS pipeline → Barge-in support — speaks and listens simultaneously → Decoupled intelligence and speaker modules — clone any voice without retraining the brain → Trained on 8M hours of audio → Chinese + English (Whisper encoder gives broad multilingual understanding) ━━━━━━━━━━━━━━━━━━━ The architecture shift that matters: Traditional voice agents: audio → ASR → text → LLM → text → TTS → audio. Every hop adds latency, loses nuance, breaks prosody. Covo-Audio: audio → model → audio. One pass. Natural conversation, not robotic turn-taking. The decoupled speaker design is the enterprise unlock — swap the voice without touching the intelligence. Custom brand voices, same brain. 🔗 Model: https://lnkd.in/g6-TDqqp 🔗 Code: https://lnkd.in/gsjbThrB 🔗 Paper: arxiv.org/abs/2602.09823 Building full-duplex voice AI agents for enterprise — call centers, customer support, multilingual? That's what we do at Zingaro AI and LiteCompute AI. DM me. ♻️ Repost if useful. Follow Pasha S for daily open-source AI drops.

Typing is dying. In 2025, your brand needs to be ready to be spoken to. Last week, I caught myself saying: “Remind me to call the agency at 11 AM tomorrow.” Then, “Translate ‘cotton kurta’ to Bengali.” It hit me—I wasn’t typing anymore. I was talking to my phone. This isn’t just a habit shift. It’s a search behavior revolution. Here’s what the data says: Over 34% of smartphone users in India rely on voice search daily https://lnkd.in/gH7uBAxt Voice queries in India are growing 270% YoY https://lnkd.in/gC8TNGvi Voice search adoption in India is 2x the global average https://lnkd.in/gjt5QiMd But most content is still written for 2015-era Google. Voice search is not just about keywords—it’s about context, emotion, and natural phrasing. The V.I.L.T. Framework for Voice SEO Here’s how to structure content for spoken discovery: V – Voice-First Content Write in Q&A form. Example: Typed – “Best air fryer” Voice – “Which is the best air fryer for a family of 4 under ₹5000?” I – Intent-Rich Phrasing Use long-tail keywords that reveal user goals. Typed – “Skincare products” Voice – “Which moisturizer is good for oily skin in summer?” L – Local & Language Optimization Build for vernacular and “near me” searches. Typed – “Mobile store Jaipur” Voice – “जयपुर में सबसे सस्ते मोबाइल कहां मिलते हैं?” T – Technical Excellence Prioritize schema markup, mobile speed, and structured data. Especially FAQ, HowTo, Speakable schemas. Why this matters: Voice queries aren’t just functional—they’re emotional. People ask with urgency, location, and context. And if your brand isn’t optimized to respond that way, you’re invisible in 2025. Have you tested your content for voice? What changed when you did? Let’s unpack what readiness for the conversational era really looks like. #VoiceSearch #SEO2025 #DigitalMarketing #LocalSEO #PerformanceMarketing #VernacularSEO #ConsumerBehavior #D2CIndia

Your customers aren’t Googling anymore. They’re asking Perplexity and prompting ChatGPT. This is a fundamental shift in how people discover brands. Three optimization layers now matter: → SEO = Traditional search engines (Google, Bing) → AEO = Answer engines (featured snippets, voice search) → GEO = Generative AI (ChatGPT, Perplexity) Master all three, and you win. Here’s why: 1. Search Behavior Changed Fast → Old way: “best CRM software 2024” → New way: “What CRM should a 50-person SaaS company use if we need advanced automation under $200/user?” → Voice searches are longer and conversational Your strategy has to match how people search now. 2. Customers Use Multiple Search Platforms A single discovery journey now looks like: → Google → Siri → ChatGPT → Perplexity → back to Google Each has different ranking factors: → SEO: Domain authority, keyword strategy → AEO: Structured data, concise answers → GEO: Authoritative, well-cited content If you only optimize for one, you disappear in the rest. 3. Different Platforms = Different Audiences & Intents → Featured snippets attract far higher CTR than standard results → SEO captures broad themes → AEO answers specific questions → GEO handles complex, multi-part requests 4. AEO & GEO Build Authority → Featured snippets sit above organic listings → Voice assistants often read directly from those snippets Winning them gives you instant visibility. GEO takes that further: → When ChatGPT mentions your company → When Perplexity cites your research It shapes perception before prospects ever hit your site. To win, you need content models can: → Understand → Cite → Reuse across thousands of conversations 5. Voice and AI Are Surging → Voice search, smart speakers, and conversational AI are all growing fast → Voice queries often carry higher purchase intent than traditional search Ignoring this = ignoring your highest-intent traffic. 6. Multi-Platform Authority Compounds When you: → Rank in traditional search → Own featured snippets → Get cited by AI tools You create a reinforcing loop of credibility. → Search engines reward consistent expertise → AI models prefer sources they see repeatedly 7. Future-Proof Your Strategy Now → Every major player is betting on AI-powered search → Younger buyers increasingly start with AI tools, not search engines Companies adapting now will dominate as AI becomes the primary discovery layer. Those that wait? They’ll be invisible to a generation that never learned “old-school” search habits. Bottom Line: Winning brands aren’t choosing between SEO, AEO, and GEO. They’re integrating all three. Your customers are already using AI tools, voice search, and expecting instant answers. The real question isn’t whether you should adapt. It’s whether you’ll adapt before your competitors do. Follow Carolyn Healey for more on AI and marketing content.