By James Quinn.
You might be thinking: is there a place for document automation now that the new, glitzy category of GenAI (or more specifically LLM) technology has arrived with overlapping use cases? The answer is: yes. But as the CEO of a document automation provider you’d expect me to say that, right? What you might be surprised to hear, however, is that I see an opportunity for LLMs to significantly enhance document automation in a way that unlocks a whole new level of benefits for the drafting lawyer. Let me explain.
Most of us will have at least some experience of using legal tech to enhance legal drafting. Despite the wide variety of options out there, the conversation, certainly in law firms, is dominated by two categories – document automation and new(er) and exciting LLM-based approaches. These categories work in fundamentally different ways and each offer distinct advantages and challenges. Let’s first explore the differences between LLMs and document automation in legal drafting, examining their respective benefits, limitations, and future potential. I’ll then move on to explain why we believe the biggest opportunity will be unlocked by using the best features of each category.
Document automation in legal drafting
Document automation is a very well-established and well understood drafting technology used by many law firms, including almost all large law firms. As you likely know, software-based automation solutions use a combination of templates (precedents) and rules-based systems to generate first draft legal documents quickly and accurately. The key here is for law firms to have confidence that all documents produced via automation rigidly adhere to a law firm’s approved drafting, conventions and house-style. In short, document automation offers a highly accurate, reliable and efficient way to automate repetitive drafting tasks – from the mundane insertion of party names to the automation of highly complex suites of documents.
Sounds great doesn’t it, and very aligned with law firm thinking on consistency, compliance and minimising errors.
…however, in order to use document automation you need the following:
1. Templates
Document automation platforms require you to have a document template to act as the base for the automation (unlike LLM based approaches). The scope of the automation is limited by the underlying template – document automation can’t provide predictive insight or recommend freeform wording that deviates from the template’s text.
Some firms have invested in centralised KM and have templates on hand (a stellar investment of time and money, well done) – but others have not.
Deal documents may provide a starting point, but they require effort to convert into an automatable template with all of the required optional drafting. Don’t throw deal documents at a document automator – who isn’t an experienced PSL and subject matter specialist – and expect stellar results.
So the first question is does your firm have relevant templates? If not, is there a business case for creating them, or do you have access third party templates you can automate?
No templates, no document automation.
If there is a template available, use it – particularly for longer, more complex documents. Why wouldn’t you want to work from a risk controlled, signed off template that is likely to have been road tested for years?
As a side note, functionality to create templates through an automatic LLM driven merger of deal documents is nascent at best. Proceed with caution.
2. Implementation
Document Automation needs to be implemented for each template, LLMs do not. You might undertake implementation in-house, use a consultant, or adopt our company’s approach (which includes full implementation and maintenance within its licence). You may also adopt standard content that has been automated in advance (for example our Real Estate Plus solution), but whomever undertakes the implementation and whatever tool is used, the requirement is still there.
Unless you are using content automated in advance, you might next consider whether there is a business case for automating your suite of documents, given that it isn’t just a case of uploading them.
Can’t the LLMs just do the implementation for us? No. At least not with the current state of the art or at any level of complexity that results in a meaningful automation (i.e. that adds value to your firm).
LLMs and a blank page
LLMs mimic creative output exceptionally well. The language and grammar output by the more recent models are truly impressive. GPT-4o can write you a decent sea shanty about legal tech – far exceeding my own sea shanty’ing efforts.
In the past, if you didn’t have template wording or a past deal document to use as a base, then your drafting started with a blank page. You might agree that it is much harder to draft from scratch than to review, so it’s often (not always) helpful to be provided with a starting point.
More often than not, LLMs provide that starting point to work from. You do of course need to be a subject matter expert to review the output given the risk of hallucination (see below), but that doesn’t prevent the generated text being helpful.
In order for the output of an LLM to be worthwhile, you don’t necessarily have to use the generated text – it is enough that the output moves your thinking forward. Even if the resulting text is only an example of how not to draft a given clause, it often moves you closer to the form of words that you do want.
I often use LLMs for inspiration or to draw out additional points / ideas. For example, I recently drafted a routine short form lease, that also required some non-routine rights and obligations relating to maintenance and parking. An LLM (in this case GPT-4o) did a passable job at suggesting an additional clause which included a number of helpful elements I hadn’t thought of.
So, using an LLM to generate drafting often beats drafting from a blank page.
Consistency and Review
Let’s talk about consistency.
An LLM-generated output is different every time. Great for inspiring original thought, but not what we want in a legal context.
Document automation does not waiver from the drafting outcome dictated by the automation logic. You get the same drafting output every time (when I say yes to a “non-compete obligation” I know exactly what is going to happen in my document). Also, the wording not amended by the automation logic is the wording of your base template. From a risk and review perspective this is absolutely crucial.
There is a real danger that LLMs save junior lawyers time in the drafting, only for that time to be frittered away in undertaking an entirely necessary head-to-toe review of the drafted output.
Worse still, if the review time of more experienced (expensive) supervisors increases because they have to review uniquely generated content, that is a terrible result.
It’s also worth noting that it’s quite challenging for humans to review LLM generated content because the language and grammar are so good. You can be easily lulled into mistaking well-structured language for legally correct content. Hard to do that correctly if you’re junior or if it’s 2 a.m. and the pressure is on. What would the PI insurers think? Best not to ask.
It’s much easier to review the drafting variation in a trusted template. On the whole, a supervisor won’t review the whole document, just the amendments to the signed-off template. For consistency and review time, document automation is the way to go.
Hallucinations
A general openness and enthusiasm for GenAI is not the same as unwavering trust in its output. In the 2024 LexisNexis generative AI survey, only 10% of respondents said they had no concerns about using generative AI, while 59% admitted to having some concerns, 26% had significant concerns and 6% had fundamental concerns.
Lots to cover here on hallucinations and I feel that I won’t do it justice in this short piece. Suffice it to say that not having a concept of truth and error is a significant problem for LLMs and may ultimately place a ceiling on progress here. Adopting a Retrieval Augmented Generation (RAG) approach really helps but does not eliminate the problem. There is always a non-zero chance that the next word in your generated output is going to stray from the truth and that, in turn, leads to a higher and higher chance that each subsequent word will also go awry (the risk of error compounds). This is less of an issue with shorter texts but the longer the output the more likely that it will drift away from truth.
There is lots of great work going on in this space but there is clearly an issue (read Artificial Lawyer here, and the articles referred to therein).
It’s also clear that if we lock down the creativity of LLMs by limiting the functionality to presenting retrieved content then we have simply created a content search engine. Nothing wrong with that, but good to know what we are dealing with.
Document automation systems don’t hallucinate. Strange to have to point that out, but highly relevant when comparing document automation to LLMs.
Is hybrid the future?
In the context of producing legal documents, LLMs and document automation share the same ambition – to make drafting more efficient. And with this common ground comes an opportunity to collaborate. This isn’t a choice between two competing technologies, it is an opportunity to leverage the strengths of both.
• Firstly, by matching technology choice to the nature of the drafting requirement. Where you have a template, document automation is almost always the preferred tool – generate a fast, consistent output that requires little review. But if you don’t, or if your template doesn’t cover your scenario, LLMs can provide some drafting. This drafting might be an extra clause to augment an existing template or a new document to act as a tentative starting point where no template is available (caution here, this will usually be a version 0.1).
As a side note, where you are mixing drafting generated by both document automation and LLMs, make sure you clearly and separately identify the source of all wording so that the drafting lawyers can apply the appropriate levels of review (and trust) to each section.
• Secondly, super-charge your automation by integrating ‘good’ drafting generated by an LLM into your automated templates on an ongoing basis. If you identify a gap or opportunity to extend your template documents, LLMs can assist you to fill that gap – perhaps a drafting lawyer generates a new arbitration clause with the assistance of an LLM. Capture that clause and feed it back into the template so that everyone can benefit going forward – subject to a thorough review! You don’t want your lawyers all asking the LLM to draft the same clause repeatedly – resulting in endless variations which all need review and sign-off.
In conclusion, for law firms to get the most out of these two rapidly growing technologies, we need to stop treating the selection of LLM-based technologies or document automation as a binary decision. The biggest benefits could be achieved through a smart combination of the two.
By James Quinn, CEO, Co-Founder, Clarilis.
—
[ Image: AL tries to avoid all genAI created ‘art’, because it’s an affront to real artists. However, as the ‘Care Bear’ parody picture sent in by Clarilis is clearly meant to be an ironic abomination, then we can perhaps give it a free pass. ]