What happens when you have too much information? Large language models (LLMs) are inherently limited by their so-called "context windows". An LLM can only process a finite amount of text before it simply "forgets" some information in order to make room for more. This is a fundamental limitation of the architecture of LLMs, and it has significant implications for how they can be used in legal applications, where source material (contracts, laws, compliance guidance, etc.) can often be very long and complex.
Furthermore, even when information does fit within the context window, an LLM may still struggle to effectively process and reason over large amounts of text, especially when the task requires complex reasoning or understanding of nuanced legal language. This is why you might get mixed results when you simply feed a long legal document to an LLM and ask it a question. The model might latch onto the wrong parts of the text, get confused by irrelevant information, or simply run out of "attention" to give to the relevant parts. You cannot fix this issue with better prompting or instructions to the model, because the problem is not just about how you ask the question, but also about what information the model has access to and how it processes that information.
So how do designers of AI powered legal applications typically deal with this issue?
This demo will demonstrate a common approach to this problem, which is to use a retrieval-augmented generation (RAG) pipeline. This demo allows you to inspect that process step by step using a preloaded EU directive as an example. It shows how choices made before the model is even called, such as chunking, retrieval, and background prompting, can significantly influence the final answer.
Current step: question
The current step indicator shows where you are in the process. You can restart the pipeline at any time.
Preloaded document: EU Directive
This demo uses one predefined legal source. The goal is to show how retrieval changes what the model receives.
This demo uses a single EU directive as the source document. It is approximately 100 pages, and while it fits within the context window of the model, even a document of this size could lead to LLM hallucinations if complex reasoning is required. In real applications, the source could be anything, such as hundreds or thousands of contracts, years of case law, legislation, or compliance guidelines well beyond the model's context window.
As the end user, this might be the only part of the system you see. You press a button, trigger an event, or simply ask a question in natural language, and the system does the rest behind the scenes.
A large part of enterprise legal tech is of course the UI/UX (user interface and experience). As the primary purpose of this demo is to show the RAG process, the interface is simply a standard question input field to let you experiment with different settings in the RAG pipeline that are usually hidden from users.