Recall 2.5: the trace view, or how to show your working

What did we concede on change-of-control in the last three private-equity exits where we acted for management?

Recall has been able to answer a question like that since 2.3. What it could not do, until today, was show you precisely how it arrived at the answer in a form you could check line by line. Recall 2.5 adds the trace view, and it is the largest change to the product since permission-aware search.

What changed

• Trace view on every answer. Beneath each synthesised answer there is a trace: an ordered list of the steps Recall took, the passages it read at each step, and the document and matter each passage came from. Each passage opens in context in the source document.
• Passage-level citations in the answer text. Each sentence in a synthesised answer carries a marker linking it to the passages in the trace that support it. Sentences with no supporting passage are not generated; if Recall cannot ground a sentence, it omits it and says what it could not find.
• Permission path per passage. The “why can I see this?” link from 2.3 is now shown inline in the trace for every passage, not only for top-level results.
• Query interpretation shown. The trace opens with Recall’s reading of the question: the entities it identified, the date range it inferred, the matter filters it applied. If it misread “management” as a company name rather than a role, that is visible before any results are.
• Trace export. The full trace, with passages and citations, can be exported as a document for a matter file or a supervising partner.

Why

Two reasons, one practical and one principled.

The practical one is that synthesised answers were being trusted too much and too little at the same time. Some users read the three-paragraph summary and acted on it. Others ignored the summary entirely and went to the result list, because they could not see where the summary had come from. Neither is the right behaviour. The right behaviour is to read the summary, check the parts that matter, and move on. That requires the working to be visible.

The principled reason is the one we set out in April: an answer without a citation cannot be audited, and legal work product has to be auditable. A partner asked, a year from now, why the firm took a position on change-of-control should be able to open the matter file and find not just the answer but the precedents it rested on and the route by which they were found.

The trace is not for every query. It is for the one query in twenty where it matters, and you do not know in advance which one that is.

What to watch for

Answers are shorter. Because Recall now omits any sentence it cannot ground in a retrieved passage, synthesised answers in 2.5 are on average around 30 percent shorter than in 2.3. Some of what has gone was useful framing. Most of it was plausible filler. We think the trade is correct, and the result list beneath the answer is unchanged if more context is wanted.

“Could not find” is now a common response. Where 2.3 would have produced a general answer, 2.5 will often say that it found passages on part of the question and nothing on the rest. This is more honest and, for some users, more frustrating. The trace shows what was searched, which usually makes clear whether the gap is in the firm’s records or in the query.

Query time is marginally longer. Building the trace adds around 200 milliseconds at the median. The permission-path lookup per passage is the main cost.

Traces are retained with the query log. This is subject to the firm’s existing audit-log retention settings. Firms with short retention will lose traces on the same schedule as the queries themselves.

Known limitations

• The trace shows passages Recall read and used. It does not show passages it read and discarded. We are considering whether to expose these; the volume is large and the value uncertain.
• Where an answer depends on a count or an aggregate (“in seven of the last ten”), the trace lists the seven and the ten, but the counting step itself is shown as a single line. If Recall has miscounted, the trace makes it checkable but does not make it obvious.
• Passages from scanned documents without a text layer are cited by page and region, not by text, and the in-context view opens the page image.
• Trace export does not yet include the permission paths, which can contain group names the firm may not wish to circulate. They are visible in the interface only.
• Very long traces, over about forty steps, are collapsed by default and can be slow to expand on older hardware.

Smaller changes in 2.5

• Saved queries can be re-run with their original date range frozen or rolled forward.
• Results from matters the user has worked on personally are marked.
• The matter filter now accepts practice group and supervising partner as well as matter number and client.
• Query latency for firms with large wall registers is improved after the work promised in 2.3; the 95th percentile is down from 3.3 to 2.4 seconds in our benchmark.

If you have been reading the result list and ignoring the answer, try reading the answer and opening the trace. It is the same evidence, with the route drawn in.