Search quality

Search stands between a user and the right agent, so its quality decides whether the whole system feels reliable. This page gives the measured numbers and the conditions they hold under. Everything below was measured against the live search service, running the code we ship with the settings it ships with. Nothing is modeled or extrapolated.

Test catalogs

Search reads a question typed in plain language and finds the agent that answers it. To measure how well it does that, we need catalogs of agents and questions with a known right answer. We built five, each a realistic working catalog for a different kind of business, plus one very large catalog to see what happens at scale.

That is 15,000 questions in total. Every question has one known correct agent, and every question was put through the real search.

The five business catalogs differ in the way that matters most: how much their agents overlap. A university's agents sit well apart; a retailer's supply chain agents share a great deal of ground. That difference decides how hard search is, and it is why we do not report a single number.

How often it finds the right agent

Search returns the single best match or the two that are plausible, never more, so two numbers matter: how often the right agent is first, and how often it is one of the first two.

Two things stand out.

The spread between catalogs is larger than the spread between any settings we tried. The easiest and hardest catalogs here are about seven points apart. What separates them is how distinct their agents are from one another. You control that when you write your agents; search cannot fix it afterwards. Describing an agent covers what makes a description distinct.

Size is not the problem people expect. A catalog of a thousand agents scores within a point of a catalog of a hundred. Adding agents does not make search worse; adding agents that overlap does.

One answer or two

When the best match is clearly ahead, search returns that one. When the top two are close, it returns both and the user chooses. Returning one more often saves the user a click, but a single wrong answer is worse than two that include the right one, so we set the line by checking every single answer against a reviewer's judgment (see below).

Returning no match when no agent fits

A search that always answers is harmful, because the wrong agent will act. We therefore test refusal as carefully as matching, with two kinds of question that should both be turned away.

Questions about something else entirely. "How to replace brake pads on a 2012 Honda Civic", asked of a university's catalog. These are refused 100% of the time.

Questions about the customer's own world that no agent covers. Asking a university about appealing a parking ticket when nobody has built a parking agent. These questions are harder, because every agent in the catalog is in the same neighborhood and something always looks close. They are refused 82% to 86% of the time, depending on the catalog.

The second case is the one that matters in practice. People do not type a sourdough recipe into a company's search box; they ask about the one thing nobody has built an agent for yet. What the user sees when search returns two agents, or none, is covered in How matching behaves.

Speed

A search compares the question against everything in the catalog and returns the best matches in about 200 milliseconds in Search2o Cloud. A catalog of a thousand agents takes about the same, a little over 200 milliseconds. Search does not get slower as the catalog grows. End to end, with the network hops, a user can expect an answer in well under a second. Search2o Cloud is served from the US East region today; other regions will follow as demand grows.

Across languages

Search works on meaning rather than words, so it handles languages other than English, and a question asked in one language against agents described in another. Accuracy in a non-English catalog is within about three points of English, and asking across two languages costs about three points more.

Limits of these numbers

Every number above comes from catalogs we built, and a real catalog will differ. Read the numbers as a range: a well-separated catalog scores at the top, a catalog full of near-duplicate agents at the bottom, and the difference is mostly in how the agents are described.

The numbers are likely lower than real use

The test questions, and the answer each is marked against, were written by a language model and made hard on purpose. Many name something that belongs to one agent while asking for what another agent does: "What are the top drivers behind incident date by severity?" is marked as belonging to an insurance-claims agent, because "incident date" is one of its fields, though most people asking it would want the incident-report agent.

A question counted as wrong is therefore often one a person would have answered the same way. We had every question where search's first choice differed from the answer key, about 2,000 of them, judged blind: the reviewer saw only the question and the two agents' descriptions, in random order, without knowing which was the key and which was search's choice. In about three cases out of four the key stood. In the rest, search's choice was as good or better. Counting those, the right agent comes first more often than the first table says:

Some of the questions are confusing enough that we would get them wrong ourselves. Real people ask what they mean, in their own words, and we expect search to do better on those than on ours. The fastest way to measure your own catalog is to add your own agents and ask your own questions.