Reasoning that truncates: when the model eats the answer's budget

For weeks SETEK looked for the cause of a defect in document retrieval, while it actually sat in generation. In a conversational assistant answering over a document corpus (Vertex AI RAG Engine with Gemini 2.5 Flash), roughly 20% of questions received what looked like a refusal — “I have no information on this topic in the available documents”. A refusal naturally points to retrieval: chunk size, distance threshold, category routing, and the semantic re-ranker were all investigated. None of it explained the remainder.

The defect was invisible because the code never recorded the “finishReason” field, which states how generation ended: without it, a truncated answer and a refusal are the same event. A datum you fail to record is not merely missing: it actively misdirects the analysis, because the defect disguises itself as another, already familiar one.

The mechanism: reasoning models produce, before the text meant for the user, a hidden chain of reasoning whose tokens are drawn from the very same output cap as the answer. Left dynamic, that reasoning expanded to 984 tokens out of 1024, leaving 36 for the answer — just enough to start a sentence and run out of room. Even among answers that ended normally, median reasoning was 559 tokens out of 1024: more than half the budget never reached the user.

The test ran on a frozen context — one retrieval per question, the same prompt byte-for-byte across every configuration — over 89 questions, 50 of which were negative controls deliberately outside the domain. By declaring the reasoning budget and adding it to the cap, truncations went from 20 out of 39 to zero, with 50 correct controls out of 50 in every configuration: the system did not become more willing to answer what it must decline.

The rules SETEK draws from this apply to any system built on reasoning models, whatever the vendor: always declare the reasoning budget, because the dynamic default takes all the room it finds; make the cap exposed to the administrator apply to the answer alone, adding the reasoning budget on top; always record how each generation ended; freeze everything else when comparing configurations; treat negative controls as seriously as positive cases; regard every cache layer as a defect multiplier, since it keeps the first answer rather than the average one; and repeat security tests, which against a non-deterministic system yield an estimate, not a binary verdict.