Horace Wu’s Syntheia has found that by taking a different approach to how a contract is ‘sliced up’ before applying AI, to put it simply, lawyers can significantly reduce their token costs. (See AL Interview below.)
The token savings are therefore not gained by switching models or ‘rerouting’ to cheaper LLMs, but by changing the starting conditions of the doc review.
The findings come as the debate about how to reduce rising token costs in legal tech widens across the market.
The company stated that: ‘In transactional legal work, the cost of an LLM reading a document is typically far larger than the cost of it reasoning over the answer.
‘In our tests, the length of the final answer barely varied between methods, while the amount of text fed into an LLM varied by up to 30×.
‘That gap is where we can find significant savings. As AI agents increasingly decide for themselves what to retrieve and read, that matters even more.
‘Syntheia’s research team tested two structured retrieval methodologies for transactional legal text, both built on our structure-aware document indexing technology, against full document injection on a 20-question benchmark spanning real credit facility agreements, limited partnership agreements, and share purchase agreements.
And here are the headline results:
- ‘Semantic (embedding-based) retrieval, which fetches only the passages most relevant to a question, matched the performance of full document injection on 18 of 20 benchmark questions while cutting tokens processed by 17.3×. A faster, lighter-weight embedding configuration pushed the token reduction further, to nearly 30×, with a modest trade off, matching full injection on 15 of 20 questions.
And,
- Structured index navigation, a novel format Syntheia developed for LLMs to navigate in order to decide what to retrieve, reasoning over a compact map of a document’s clauses rather than the full text, matched full document injection performance on all 20 benchmark questions, while cutting the context the model reads to compose each answer by roughly 56× and total tokens processed by 1.6×.’
In short, by focusing on what is likely to be most relevant you can massively cut down on token use.
AL Interview with Horace Wu, founder of Syntheia.
What models did you use to achieve this?
This research was started two months ago, so it was Claude 4.6 as the main engine for Q&A.
Tell us about the use case.
This isn’t targeted at research, but at Q&A over transactional docs.
The use case is where law firms (and GenAI companies) have to look over contracts to answer questions, e.g. ‘what are my termination rights here?’ The method we have published shows cost savings and token savings.
And how do you keep accuracy and use fewer tokens?
The savings are from cutting unnecessary tokens from the context window + removing the need to do double, triple passes over the document to pull defined terms, etc. This is possible because of how we structure the document beforehand and create the links between clauses and definitions.
Our method maintains accuracy. LLM accuracy for legal documents is already in the 90s, and so a few points of uplift is hard to achieve (and also needs a LOT more experiments to prove). What we are trying to prove is that accuracy does not degrade even as we cut tokens from the context window, because we are providing BETTER tokens to the LLM.
And in terms of RAG, which you mention in the paper, how does that apply here?
Oh… I can talk RAG all day. So, basically, the hardest part of RAG is deciding what to ‘retrieve’ in the first place. For 99% of the world, they are using vector embeddings to do retrieval, i.e. I have a question or a search term, go and find for me something that means the same as this (semantic similarity).
Vector retrieval is powerful, but it has problems. What we are doing here is we are doing retrieval by letting the LLM reason over a short index of the document.
Syntheia creates this index, which is like a table of contents, and the LLM reads this table of contents with the question and it goes, ‘Oh, I need to retrieve clauses 3 and 7 to answer this question.’ So, it doesn’t use similarity search at all. It uses reasoning.
What are the savings benchmarked against?
Savings are benchmarked against full injection of a whole document.
We also tested it against vector RAG, and showed the difference in token spend and accuracy. Short version – vector RAG gets people around 90% of the way there in terms of accuracy, but it is much cheaper.
We can get all the way there, and still give a significant saving. A caveat though – the research is early, and we have plans to go much deeper very soon. Part of that future work is a new benchmark built on top of LegalBench-RAG that adds the layout information recovered from the CUAD source PDFs, which most text-only retrieval benchmarks discard. Putting this initial version on arXiv now timestamps the ideas that led here and makes sure our work is credited.
How should law firms handle token costs? Should they pass them to clients?
Depends (such a lawyer answer). Law firms who have signed with Harvey and Legora should go with all-you-can-eat for as long as they are allowed. For law firms who are building on Claude or have to otherwise pay for consumption, then it is about minimizing token spend while they ensure there are still quality outputs.
As to whether they should pass the spend onto clients, I think there will be a lot of resistance, and that’s a commercial discussion most folks haven’t had yet.
Do you think showing token savings will become a major component in legal tech sales pitches?
Yes, especially because lawyers will want to make sure AI does the work properly and to the best possible standards. The cost of manually verifying AI outputs is enormous and unpleasant.
If we can get the AI to push out the best possible output and also indicate where they are uncertain (plus other systems to flag potential errors), it will improve the whole point of using AI. And if the idea is that AI is supposed to help deliver better legal outcomes, then improving those outcomes at a reasonable price is important.
And here is the Arxiv paper by the researchers.
More about Syntheia here.
—
Come and join us in New York and London this November at Legal Innovators!
Legal Innovators UK – London, Nov 4 and 5
And, then Legal Innovators New York – Nov 17 and 18.
After another fantastic Legal Innovators California, where we had speakers from OpenAI, Y Combinator, Google, Meta, and many more pioneering organisations; and our stellar inaugural event in Paris this June, we are now looking forward to the landmark conferences in London and New York, both in November, and both across two days: Law Firm Day, and Inhouse Day.
Discover more from Artificial Lawyer
Subscribe to get the latest posts sent to your email.