AI engineering · 2026-10-18 · 5 min

A larger language model gave worse answers. Measure it on your own documents.

For an AI chat over a fund manager's data room we compared several language models on the same 39 questions about the real fund documents. The smallest model in the comparison scored highest. The larger, more expensive models fell behind. The lesson is not that small is always better, but that choosing a model is a measurement, not a feeling.

The measurement

We put together a fixed set of 39 questions investors ask in practice, across eleven topics: fund basics, fees and waterfall, strategy, team, track record, governance and reporting, risks and pipeline. Added to that: follow-up questions without context ("and how is that split?"), three questions whose answer is not in the documents, and three attempts to make the chat do something it should not.

Every question has fixed requirements that can be checked automatically. They are deliberately boring:

  • The answer states a specific fact from the documents.
  • The answer cites one of the documents that contain that fact.
  • If the answer is not in the documents, the chat says so and cites no source.
  • If someone asks for a private phone number or the system instructions, they do not come out.

We ran every configuration three times, at temperature 0.

What came out

  • The smallest model: 36, 36 and 37 out of 39.
  • Two larger models: between 30 and 33 out of 39.
  • For comparison: the very first version of the chat, before we tightened the search step and the instructions, scored 29.

The larger models did not fail by making things up. They more often left out a fact that was in the documents, and they attached sources to answers that should have said something is not in the documents. To an investor both are errors: an incomplete answer, or a source suggesting something is backed up when it is not.

Why exactly, we do not know for certain, and we will not fill that in here. What we do know: with the same search step, the same documents and the same instructions, the smaller model more often gave the complete answer with the right source.

Temperature 0 is not the same as the same outcome

Same model, same questions, temperature 0, three times in a row: 32, 33 and 30 right. A difference of three questions out of 39, with nothing changed. Whoever tests once and compares one score is choosing by chance. Run every configuration several times and look at the spread, not at the best number.

How to set it up yourself

  • Write down the questions users actually ask, including sloppy phrasing and follow-ups. Around forty is enough to see differences.
  • Include questions whose answer is not in the documents. A chat that makes up an answer there is more dangerous than one that sometimes misses something.
  • Make the checks mechanical: which fact, which source, when no source. Do not let another language model grade the answers if you want to measure the difference between models.
  • Run every configuration at least three times and keep every run. Then you also see whether a change improves anything or is just noise.
  • Keep the failed questions in the set. That way every next change is held to the same bar.

These numbers hold for one data room, one document set and one way of searching. This is not a general benchmark of language models. That is exactly why you should measure it on your own documents.

PRODUCT
AxonVault

AxonVault is a data room for fund managers with AI Vault Chat, tested against fixed question sets on real fund documents.

View AxonVault →