LITIG, the Legal IT Innovators Group, with support from Artificial Lawyer, has formed a Legal Industry AI Benchmarking Collaboration initiative in order to address the growing need for shared standards in the use and measurement of generative AI tools.
LITIG is calling on all interested parties to get involved, with a first in-person meeting to take place in London later in July to begin the project. If there is international interest an online meeting will be also convened afterward.
If you would like to participate, please contact LITIG via the online form here.
And now, to explain more about the project is John Craske, Chief Innovation & Knowledge Officer, CMS, and a Board member of LITIG, who has created the initiative and will help to host it.
—
‘As the legal tech world gets to grips with generative AI, building trust and mechanisms that promote transparency are going to be critical for success.
The publication of the recent Stanford University report: ‘Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools’ and the ensuing debate by interview, article and press release has been really interesting. It can’t be true that the same tool is both 90% accurate and 42% accurate!
At CMS we have been doing our own testing, benchmarking and gathering data about how accurate generative AI solutions are. I know many other firms are doing the same. For example: Linklaters did some fantastic early work in this area (see the LinksAI benchmark); Ashurst have recently published some qualitative data and information about their testing approach; and Addleshaw Goddard have explained aspects of their approach to testing methodologies to boost accuracy. In recent news, legal tech company Screens has also published their generative AI accuracy scores and methodology (see Artificial Lawyer piece on Screens).
For me, this is a fantastic opportunity for us to come together across the legal industry and discuss the issues and – more importantly – to agree a way forward.
My own view is that we need to, at the least, agree on some standard tests and a methodology so that we have a level playing field.
There are already some standard ways of measuring AI accuracy and we should consider using these. Imagine a similar approach to ‘Miles Per Gallon’ figures when you are comparing car performance. This and other topics, such as whether we also need to have a way for tests to be independently verified, what tests / use cases we need to include in the benchmarking, and how the benchmark would be maintained, are things that would be great to discuss with a broad, representative group across the legal industry to get some consensus and buy-in. I am sure there are other things to consider as well.
LITIG is therefore today launching a call for anyone in the legal industry who is interested in collaborating on this to register their interest using this form.
We are interested in gathering views from people in the legal industry with knowledge and experience of generative AI, at law firms, suppliers, regulators and representative bodies, academics and in-house teams. The aim is to arrange an in-person session in London, so that we can conduct some group work, possibly in late July, in the CMS offices. Of course, we can’t host hundreds of people, but we will aim to be as inclusive as possible and publish the output of anything we do in collaboration with Richard Tromans and Artificial Lawyer and at Litig.org.
Because we are interested in hearing from people based outside of the UK, we will hold open the option of running an additional online event for people who cannot get to London.
We look forward to hearing from you.’
John Craske, Chief Innovation & Knowledge Officer, CMS
—
Note from Artificial Lawyer: if and when other industry benchmarking groups emerge, the LITIG AI Benchmarking project will be more than happy to share its results and engage constructively with them. This initiative is for the benefit of everyone.
P.S. as noted, if you’d like to participate, please contact LITIG via the online form above. Artificial Lawyer is providing support to the project, but is not organising it.