Sales LLM: Maya Beats GPT-6 Luna on 471 Real Sales Calls

Sales LLM: Maya Beats GPT-6 Luna on 471 Real Sales Calls

A Sales LLM Beats the Frontier Models, and the Benchmark Is the Real Story

SquadStack.ai has launched Maya, which it describes as the world’s first Sales LLM. On the SquadStack Telesales LLM Benchmark, Maya scored 96.7 per cent and beat ten models from OpenAI, Google, DeepSeek, Alibaba, NVIDIA and Indian AI labs. The benchmark asked eleven models to write the next agent reply on 9,962 moments drawn from 471 real calls, and a neutral judge, Anthropic’s Claude Haiku 5.5, scored each reply Pass or Fail against seven checks covering matters such as staying on the call’s goal and using tools correctly. Google’s Gemma 4 31B scored 92.5 per cent, DeepSeek V4.1 Flash 92.2 per cent and OpenAI’s GPT-6 Luna 89.0 per cent.

Maya begins its reply in a median 278 milliseconds on a production endpoint in India, with 95 per cent of replies starting within 774 milliseconds. SquadStack post-trained the model from an open-weight foundation on more than 40 crore real sales interactions gathered over six years, each tagged with how the call ended. It handles more than 50 lakh conversations a day in Hinglish and Indian languages. The release gives no human baseline against the judge, no inter-rater agreement, no confidence intervals, no named customer, no conversion lift and no pricing.

What makes a sales call hard for a model

The release opens with examples that are worth taking seriously, because they name the actual difficulty rather than a generic one. A customer says haan theek, and the agent cannot tell a yes from a polite acknowledgement. A borrower says sochke batata hoon, and the agent either pushes too hard or lets the lead go cold. And nearly half the time the agent replies to speech it only half heard.

Those are not language problems in the sense a benchmark like MMLU measures. They are problems of intent under ambiguity in a code-mixed register, where the same short phrase carries different meanings depending on tone, context and what the customer said three turns earlier. A general-purpose model trained predominantly on written English has no particular reason to resolve haan theek correctly, because nothing in its training rewarded that distinction.

The fourth difficulty is timing. A sales call is a synchronous conversation and the customer hears every slow turn. That makes latency a product requirement rather than an engineering preference, and it is the constraint most model comparisons ignore.

The benchmark, and the judge inside it

The more durable contribution here is the benchmark rather than the model, because no public benchmark measured what a language model does on a live sales call and SquadStack built one. Its design is worth examining on its own terms.

The structure is sound. Real calls rather than synthetic transcripts, a defined moment where each model writes the next reply, seven specific checks rather than a single quality score, and a judge from a different vendor than the models under test. Using Anthropic’s Claude Haiku 5.5 to score OpenAI, Google, DeepSeek and Alibaba models removes the most obvious conflict, and publishing every model’s verdicts on Hugging Face with a route for other teams to submit their own is more than most vendor benchmarks offer.

What the design does not establish is whether the judge agrees with a human. A benchmark scored by a model is only as good as that model’s judgement, and the release gives no inter-rater agreement between Claude Haiku 5.5 and a sales manager listening to the same call. That is the missing validation, and it is the difference between a benchmark that measures sales quality and one that measures similarity to a particular judge’s preferences. The gap matters most at the top of the table, where Maya’s 96.7 per cent sits 4.2 points above Gemma’s 92.5, and where a systematic preference in the judge would be enough to reorder the ranking.

The arithmetic elsewhere holds up. Maya failed 3.3 per cent of checks and Gemma failed 7.5 per cent, which is the more than twice as many the release claims. That is a real gap rather than a rounding artefact, whatever the judge’s preferences.

The data moat is the actual argument

The claim that carries the most weight is not the score. It is the training set: more than 40 crore, or 400 million, real sales interactions collected over six years, each labelled with how the call ended, whether a sale, a callback or a drop-off.

That is a dataset nobody can scrape. Apurv Agrawal, co-founder and chief executive, puts the argument plainly when he says anyone can download a model and nobody can download six years of real sales conversations and what happened after each one. The position is correct and it is the strongest thing in the release, because it describes a moat that does not depend on the model at all. Open-weight foundations are available to every competitor; the outcome-labelled corpus is not.

It also explains why a smaller model can beat a larger one on a narrow task, and why a Sales LLM trained this way is a different proposition from a general model with a good prompt wrapped around it. Vikas Gulati, co-founder and chief technology officer, says the sales behaviour sits in the weights rather than in a long prompt, which is the technical version of the same claim. A prompt tells a general model what to do; post-training on outcomes teaches it what worked.

Latency is where the claim gets stronger

The performance figures are more impressive than the quality figures, and more carefully reported. A median of 278 milliseconds to begin a reply is fast for a production model serving Indian traffic, and the release gives the tail as well as the middle: 95 per cent of replies start within 774 milliseconds.

The tail is the number that matters. A sales call does not break when the average turn is slow; it breaks when one turn in twenty takes three seconds and the customer talks over the silence. A Sales LLM that reports only a median is reporting the half of the distribution that never causes trouble. Reporting a p95 alongside a median is the correct way to present this and it is unusual.

The comparison is also fairly disclosed. Gemma 4 31B was measured through a shared hosting provider at 2.3 seconds for the 95th percentile, and the release states that plainly rather than implying a like-for-like contest. It then adds the comparison that is like-for-like: the models that matched Maya’s median speed on dedicated infrastructure passed only 66 to 72 per cent of checks. That is the honest framing, and it strengthens rather than weakens the case.

Hinglish is the underrated part

One detail deserves more attention than the release gives it. Maya runs more than 50 lakh, or five million, conversations a day in Hinglish and Indian languages, which is a claim about a capability that general models handle poorly and that few benchmarks test.

Code-mixed speech is hard in a specific way. The same sentence switches script and language mid-clause, the register is informal, and meaning depends on particles that carry no direct English equivalent. A model that treats haan theek as an affirmation will mishandle a large share of real Indian sales conversations, and the failure is silent because the transcript looks correct.

That also connects to the compliance point the release makes. Because Maya is built and hosted in India, customer conversations do not leave the country, which simplifies obligations under the Digital Personal Data Protection Act for banks, lenders and insurers. For a sector handling loan applications and health questions, data residency is not a preference and it is the argument most likely to close an enterprise deal.

What is missing, and one gap matters more than the rest

The release is more transparent than most vendor benchmarks and it still leaves several things unstated: a human baseline against the judge, inter-rater agreement, confidence intervals on the scores, a description of how the held-out test calls were separated from the 400 million training interactions, named customers and pricing.

The gap that matters most is the one closest to the company’s own pitch. SquadStack describes its agents as outcome-driven, and the release measures reply quality rather than outcome. A four-point improvement in pass rate is a meaningful engineering result; whether it produces more sales, shorter calls or lower acquisition cost is the number an enterprise buyer actually needs, and the release does not give it.

That is a fair thing to ask for and an easy thing to supply, because the company already tags every call with how it ended. If Maya’s deployment correlates with a measurable change in conversion against the previous model, that comparison would be more persuasive than the benchmark. The benchmark shows Maya writes better replies. The conversion figure would show the replies are worth paying for.

Sales LLM: Maya Beats GPT-6 Luna on 471 Real Sales Calls
Photo by Growtika on Unsplash

What to watch

Whether a human agreement study appears, because a benchmark judged by a model needs to show that its verdicts match a sales manager’s before anyone treats the ranking as settled. Whether other teams submit models to the published benchmark, since a leaderboard with one regular competitor is a press release and one with several is an institution. And, whether a conversion comparison appears, because reply quality is the means and sales is the end. And whether the learning loop that watches live calls and carries fixes across campaigns produces a visible improvement in the published scores over time, because a model that gets better every week should show it on its own benchmark.