Accountability, Defensibility, and the Future of AI in Discovery

Daniel, so happy to be on the show. Thank you for the invitation.

It's a big question. As you've said, no softballs at the start of the podcast, so I appreciate that. Let me just play this back to you in terms of the question. There's a lot going on in legal technology right now.

Some of the largest proposed IPOs in the world are coming down the pipe from Anthropic and OpenAI. Anthropic has signaled extremely intensely that they are very interested in legal. They've set development streams. They've set up repositories on GitHub full of skills.

They've set up partnerships and connectors between many organizations, including Relativity. So the question is, I think, are we headed in a cohesive direction or not? And I would argue that we are, but we are doing so in an unreliable way, which is to say that there are still gaps in the technology available to people today. There are still gaps in our legal frameworks in government, in education, and enablement.

So I think there are people in the top, let's say, 5% of training and education and enablement and practice in legal technology.

And you would think that they all think that we're going in one direction, and I can tell you that we don't. Let's assume for the purposes of this podcast that I made that 5%. I can tell you that other people that I talk to that are applied science or AI leaders at other large companies and organizations, I think we see some common ground in some of the investments. I think that a lot of people that say, for example, that agentic research is going to be very important.

And then in other places, we have filed disagreement. So if you talk to I don't wanna name him because he I don't know if he's publicly stated this before, but there is another applied science leader in another large legal technology organization. And he and I actually disagree very much on how much software engineering will change and whether software engineering will effectively disappear. And and that's indirectly related to legal technology, but it's directly related in the sense that, hey.

There's all this software that's already been built. Does that software still have value in a couple of years? And I would say it has a lot of value. And the specification and the intent and the training around that, in addition to the code itself, are all pieces of value that a company can continue to durably invest in.

But just coming back, are we all in alignment on what's gonna happen? No. And I think that I have a vision for how Relativity will continue to develop and build AI systems that I feel has been consistent, but I don't necessarily believe that the rest of the industry has signed up for my vision yet and that we're all rowing the boat in the same direction.

That's a dangerous question for me to try to answer because I'm not a lawyer, and I don't have a legal opinion here. And I think that ESI protocols are largely negotiated between, you know, producing and requesting parties. So just stepping back as an observer and then as opposed to sort of, like, a direct influencer here in this landscape, my observation is that there are processes within legal that are very, very rigid. We have adversarial processes.

We have places where the requesting party has subpoena authority and therefore can can sort of dictate how things are going to go. So in many of these situations, slow moving is actually correct. It allows sort of both sides to protect themselves based off of the methods and processes that are well established that deliver the right information to the right people. So things like the Sedona Principles, sort of determinations around things like undue burden or proportionality, I think those will all continue to apply.

And I believe that within the scope of a sort of request for production or another sort of highly defined disclosure process, we will continue to see steady and slow but not massively disruptive changes.

Where we will see disruptive changes is just on the periphery, where the adversarial process or the guidelines from a subpoena or some other thing do not apply.

And there's more latitude given to the attorneys and legal technology providers who are fulfilling these requests.

And in those situations, we will see much more disruptive change happening much faster.

I think it's helpful to remember that I've been in legal tech for almost five years now, and so Stockholm syndrome has set in. I admire everything around me. I don't encounter these fences anymore out in the field and wonder why those fences are there.

I think a lot of this 99%, I think of the pain that comes in the application of technology to law comes from a lack of education and enablement and systems that are designed to reliably help people get the work done and all of the things around that. It's this unreliability of the technology or a misunderstanding of the technology or a misapplication of the technology that that causes so much of the pain in sort of people's day to day. And so when I see something that doesn't make sense, a friend of mine from grad school asked me on LinkedIn. Why are all these attorneys continuing to cite these hallucinated cases?

It's like, he's like, I don't get it. Like, haven't you figured out that these things will just hallucinate citations? I'm like, yes. And think about the fact that seventy percent of these litigants who are filing hallucinated briefings are pro se.

And so they're untrained, and they're using ChatGPT. And of the remaining 30%, including some of the attorneys who have recently been sanctioned, I think fundamentally, it's a misunderstanding of the risk they are taking when they ask the system to do something without checking. I think it's a lack of education. I think it's a lack of an enablement.

There's a lot of reasons why people are not trained yet, why they're not enabled yet. I don't think the software technology is completely caught up in many situations. We do our very best in Relativity to make our technology reliable, but I don't think that that's necessarily a unified vision across the legal technology landscape right now.

This is great. So I think the study started three years ago. That's when we started, and then I think we published it in early 2024. Is that right?

2024. Yep. 2024. So our ambition was this.

We had been experimenting with GPT-4 at the time, and we were pretty convinced that GPT-4 plus document text plus something like a review protocol was on par with humans.

And so the Beyond the Bar paper and remember, there was a reference to GPT takes the bar exam where GPT was in, like GPT-4 is in a very confined space, and it was like, hey.

Take these exam questions. And now we actually wanted to see if GPT-4 applies for a real world practical scenario and situation, could do the same thing. And so we recruited five different law firms to help us in this, and I won't name any of them. There were a couple of other entities as well that were in the process of producing documents, and we were really excited about the idea that we were gonna go through and do it in a live production. And the reason why we wanted to do it in a live production is it's very hard to get high quality labels unless something is at stake. For example, the certifying attorneys production.

What we didn't do, which was unfortunate, was get and this will, again, lead to some of why the study was delayed and some other things, was the attorneys didn't go back and review the conflicts between the original first pass coding and the aiR for Review coding for several months. And when they did, they realized that the first pass review had missed a bunch of documents.

And that put them into a weird situation and put us into a weird situation. First of all, they didn't wanna talk about it. They couldn't put their name on a study when they had just kind of demonstrated to themselves that they had missed a bunch of documents in the first pass review. And so you'll notice the study doesn't include the name of the law firm or the end client and the data that was involved. Coming back to what your question was, like, were we following the scientific method? 100%. Right?

We were trying to prove or falsify the claim. Every claim must be falsifiable. Prove or falsify the claim that a generative AI approach to reviewing documents and first pass review could be on par with human reviewers. And what we found, of course, was the aiR for Review found nearly twice as many documents as the human reviewers did.

It was, like, 80% recall versus, like, 40% recall.

There's some sacred cows in discovery. One of the sacred cows is that linear human review is 100% recall and precision.

I think that the easiest way to think about it so accuracy is not a good number or agreement is not a good number to use when reporting results from discovery because, typically, discovery is about finding something uncommon and making it available. So the first thing you wanna explain in recall and precision is say, why are you using accuracy? And the reason is if I have something that only shows up 1% of the time and I say, there's none of that in this corpus, I'm 99% accurate. Right?

I've got 99% agreement with the true labels. So that's a bad metric to use. So the correction that people apply is how often do you find the thing that you're looking for when it's in front of you, and effectively, many false positives do you have? So the thing that you're looking for is recall.

So if there are 10 of these things and I find five of them, then my recall is 50%. Coming back to the example we just had, if I said that the first pass reviewers had a recall of 40% and the aiR for Review system found 80%, what I am saying is that if there were 10 responsive documents or let's say a 100 responsive documents, the human reviewers found 40 of them and the AI reviewer found 80. That's like a great easy to understand. Okay.

Here's what I was looking for. Here's how much I found. It's commonly accepted that when you're using the TAR classifier or active learning, you're only going to hit about 70 or 80% of the content that you're looking for. You will typically have 70 or 80% recall.

But there's another truism or thesis that is carried around that the human reviewers will find 100% in a linear review. That's, like, not frequently challenged that the human reviewers can miss things as well.

Let me define precision since I left that on the table. If the AI system says, I don't wanna miss any documents, and it says, all of these documents are responsive. Well, yes, you will get a 100% recall. That's guaranteed.

But then you will have a number of false positives. So precision is really about just measuring your false positive rate. It's over the total number of things I said were responsive, how many of them actually were responsive to the claim? And so coming back to our example, the AI system identified 80% of the responsive documents, but it also identified a large fraction of documents beyond them as responsive that were not.

And so I'm not gonna be able to cite the exact number, but I think it was in, like, the 30% range of precision, which is to say that one of three things that the model thought was responsive was actually responsive. And I think the humans were a good deal better. I think they were in the 50 or 60% precision situation. And, again, that meant that if they said, hey.

These 100 things are responsive, 60 of them were actually responsive. So precision and recall, they're common metrics. The one thing that a lot of people get caught on or don't realize is that precision is dependent and directly coupled to what's called the richness of the matter. And the richness is the fraction of documents in the matter that are actually responsive.

And if you have any kind of false positive rate from either your classifier or your human and a false positive is basically this misidentification that something is responsive when it isn't, the richness will determine your precision. Suffice to say, if I have a really high richness, I will almost by default have a high precision so long as my recall is good. If I have a really low richness, like, let's say, 1%, and this does happen pretty commonly in discovery. If you have a really low richness like 1%, a 20 or 30% precision is actually still pretty good, and that's because the low richness actually drags the precision down as a direct multiple based on the false positive rate of the system.

Yeah. Thanks. I will just sort of comment that the study authors are Robert Keeling, Ray Mangum, Eli Nelson, and Kevin.

It's a really great group. We were in contact with them as they ran this study. It really was their study that they ran independently. Before I comment on, you know, why it was important and how they chose the protocol, I'll sort of say one more thing, which is I think really important for table setting, which is this is not a comparison of aiR for Review against active learning. I think that's the wrong way to frame it and to think about it. It is the comparison of an attorney using aiR for Review and a managed review team that was given active learning.

Yeah. Because the human component is really important, and it drives everything that happened here. Redgrave could have hired a crack managed review team, and they could have gotten a 100% recall and precision. And, similarly, they could have done a terrible job on prompting the aiR for Review system and, you know, not done good iteration and gotten a really bad result.

So I don't wanna say, hey. It was aiR for Review versus active learning. That sort of removes the humans from the system. It's like, no.

It was a human using aiR for Review, and some humans that were in a contract review situation that were asked to review a large number of documents in a very short period of time. And that really does matter because nobody's gonna say that humans can't identify documents with really, really high recall. Of course, they can. We they used expert labels to sort of decide who was, you know, which documents were really important, but the expert labels were, of course, much, much, much slower and more deliberate and took a lot more time and required, you know, the subject matter expertise of the person who was standing in.

And that's, I think, kind of a really important thing to point out is, like, we're not saying or I don't think that Redgrave is saying and then this is, I think, one of their key takeaways is, like, it's not that this is about aiR for Review versus active learning. I think trying to take that away from the study is not really the point. The point is we know that there is a defensible process out there, which is active learning. They were interested in understanding could Relativity aiR for Review also produce defensible results.

And I think the answer is yes. And I think what it also showed that you could save a lot of time and that you could improve the recall. Because one thing that did come out of the study was that the recall and, again, this is one of the kind of key takeaways. The recall of the human reviewers equipped with active learning ended up only being about sixty four percent, whereas the recall of aiR for Review in the same set of documents with the same expert labels was eighty eight percent.

Those are the point estimates. They were taken with a ninety five five sort of margin of error. So the point is that there's a statistically significant improvement between aiR for Review over active learning, but it almost doesn't matter. It's like, okay.

Well, somebody could argue that the team didn't do a good job or the active learning algorithm wasn't correct. It almost doesn't matter. The the bigger point is that aiR for Review is a defensible system that saves a lot of time.

This is the best part about being an applied scientist at Relativity. My job is to deliver as much reliable value as possible, and then there are other people elsewhere in the company that worry about marketing and pricing and packaging and costs.

So, you know, I do know, like, you know, that I have to maintain a budget in some sense. Like, I can't do unreasonable things with the large language models or use them inefficiently, and I try to use them as efficiently as possible. But my job is to provide the value. And so there's somebody else somewhere in Relativity that can help walk you through the calculator of cost savings. I can tell you that there were over 1,100 spent across twenty four hours, And I think that most people can kinda do the math on how much a contract review team cost to do about 45,000 documents, including, you know, kind of, like, training and setup and all that. And that the one attorney I won't comment on whether this was a small amount of time or a long amount of time, but the one attorney who did the prompt iteration spent eighteen hours getting the prompt that they wanted to run through here for review.

Right there is an interesting question for you about time and savings and cost, etcetera.

I think that if you are intending to save two months of review time and effort that you can put in a couple of days, making sure that you got the prompt right. And, again, this one was a hard one because of the nuance and the request. But in terms of what do I care about and where am I focusing and spending a lot of time and energy, I believe that the eighteen hours does not need to be eighteen hours as well. I do believe that we can become more efficient in translating the intent of the attorney to a set of instructions to the model that become its actions on the documents, I think that we will see more sophisticated systems arise than just an attorney kind of typing in to the system.

So, for example, in Relativity right now, if you use aiR for Review, you don't have to start with a blank slate. You upload a memo or a complaint or a review memo, and that will automatically be translated into review instructions for the model. That's one facet that we've already sort of added. There's also a really incredible iteration experience where you can label some documents ahead of time, and you can adjust your prompt and see how the adjustments to your prompt affect or change the outcomes that you're expecting.

And so that's really important to have that interaction, but we're not done yet. I think that there's certainly more that we can do that will improve the value. I can't really talk too much about what it is, but there's certainly room here to continue to improve and to effectively take more of the information about the attorney's intent about what constitutes responsive or not and ensure that that is communicated well to the model. And I it goes beyond prompting.

Yeah. Eric, that's a really fascinating question, and there's just so many facets to it. I'm not gonna do it justice in the short answer here. I do think that there's a real responsibility of the providers and the scalers to address the needs of tokenization sustainably in the sense that we have lots of inference requests coming from around the world. And I really would prefer that we find ways to build data centers in ways that we're consuming renewable power sources. We're not contributing to high carbon emissions and things along those lines. I think that's a very responsible position to take.

I think that the users and consumers from both the legal technology, the vendors and the software developers, and the users also have a responsibility to use tokens responsibly. So to come back to that, there's this trend of token maxing where you pay for a $200 a month subscription. And if you don't use all of your tokens for the month and gather the $8,000 of API value that you were getting that month, you've somehow wasted this. And, of course, the tokens refresh, I think, every five hours.

So you can do the math on what that does to people's sleep schedules and things like that. I have seen it. I've observed it for the last six months. There is a growing anxiety, and the anxiety is different in every sort of almost, like, generation.

What is, of course, is I think maybe an inverse of the typical technology adoption curve is that the older generations are in some ways the most excited about adopting it and sort of what I would call using it in you only live once or YOLO mode and really just trying it out and hacking with it and trying new things.

And then there's an entire younger generation who's developed a very severe allergy just to the concept of AI or the use of AI, and I try to be very cognizant of that. Of course, in a legal technology company, we are AI forward. We're AI native. People who come to work at Relativity are expected to use AI as part of their jobs to make the processes better, to make our software better. We build AI products, but I'm also cognizant of the fact that that's not for everyone, and there are people who are very concerned about the implications of AI. I think, you know, again, the construction of the data centers and the responsible use of data centers is, like, a real concern, and it is something that, you know, we talk to our partners about. The other concerns around mental health and things like that, like, I definitely see it.

I see there's a disruption in what people think sort of, like, the day to day should look like, and there is sort of a reorganizing now as people try to figure out, well, what's the most efficient use of my time, or how much should I be investing in these technologies, and what's my new output expected to be. I personally am more of the mentality of well, you only have a finite numbers of highly concentrated time you can put in a day, and you should use those effectively. But that doesn't just because the AI system is happy to respond to your request twenty four hours a day, it doesn't mean that you need to be taking care of it and sending in instructions at 11PM at night and waking up at three in the morning to check on how it did. I just don't think that that's a great way to work.

It's like a $1,000,000,000 question, Dan. You all know know easy ones.

Okay. Let me take it a few ways. I think that everybody has a responsibility in this sort of, like, chain going from sort of, like, hyperscaler to vendor to legal service provider to law firm and possibly to an end client. There's a long chain here, and, ultimately, there's a point where you put your name on something and you sign it.

And at that point, you're taking responsibility.

And so the hyperscalers have a responsibility to us. They're protecting data. They're keeping their model versions consistent. They're making them available.

And in many ways, we replay those promises to our legal service provider network. If you're using RelativityOne and you run a job in aiR for Review on Monday, you are going to run through the exact same model on Tuesday and have defensible results and consistent results. And that's our promise to you. You don't have to worry about that.

We'll tell you if we're making a change, and you get plenty of time to find out that the change is coming and time to prepare for it. But then you as a legal service provider then working on behalf of a law firm or an end client, you will have a responsibility too when you use the AI system that you validated the results, that you showed them what your validation and other processes were, that you maintained their data privacy, etcetera. I don't think AI actually massively disrupts this. I think it introduces new work and new responsibilities for everybody along the chain, but those changes existed in some sense, and all those responsibilities existed before AI.

We've just added new responsibilities for everybody. And I can't say the change management one enough times because it's something that if you're just using sort of like a one off AI system and you just have, like, a, you know, a request that you type into your chat system, and one day you're using Opus 4.6, and tomorrow you're using 4.7 or Fable, you're validating everything that comes back. And so when you switch, it's a low cost switching exercise.

When you spent eighteen hours iterating on your prompt and making sure that it worked just as you intended, when we switch from GPT-4o to another model and we make an upgrade, you're gonna notice the model behavior will change. We can't preserve model performance across families like nobody can. And so, well, the best thing that we can do is ensure that you're aware that these upgrades are coming, that we're making those changes, that we own those changes, and we're gonna help you manage through them reliably. In discovery, that's a really important promise.

Thank you so much for having me.

Thank you.