Robert Keeling on Generative AI, Prompt Discoverability and Human Judgment

Thank you for having me. Really appreciate it. I'm looking forward to the opportunity to talk with you today.

So I would say it's evolved. I don't know if shifted. But essentially what we were looking at five years ago is there was kind of broad calls that at the time, TAR, Technology Assisted Review, based off various machine learning algorithms, was replacing kind of human review. And what we did is we kind of challenged that, basically, and looked at our actual matters. Because it's one thing to do a bunch of testing on, people are still testing on the Enron data set decades later, but to test on self-contained data sets. But in my practice, I'm working with clients every day on real legal matters, litigations, second requests, CID responses and the like, and we're employing technology and all of those. But what we're seeing is the limits of that technology.

And then necessarily how I think it was Elon Musk observed at one point that I think the quote was something like humans are underrated, or it turns out humans are underrated, that you will be surprised about all the ways that technology, even advanced as it is with Gen AI has its limits. And so the question isn't whether attorneys are going to be replaced or not. But the question is, is how attorneys can leverage this technology going forward.

Yeah, again, that's a great question. So what I would... advise for people who don't kind of... who are coming to this now... is that you really need to, when it comes to defensibility of process and validation with Gen AI, you really need to separate it into outward-facing uses of Gen AI versus, inward-facing uses of Gen AI. And what do I mean by that? By outward-facing, I mean. using Gen AI to identify responsive documents, but more importantly, to identify non-responsive documents. And so that even if a document set is otherwise within scope, let's say it fits the negotiated timeframe and custodians and the like, that you're using Gen AI to say, hey, look, 99% of these docs are not responsive to these doc requests. And so they're not going.

to be in the review population and they're going to be excluded from here on out. As with maybe, I think that's the primary outward facing. There's a few others, but that's the primary one. And for that, when you're really leveraging the technology to say, hey, look, there are documents that maybe hit on agreed upon search terms, but we're not going to have a human look at them because the Gen AI is saying that they're non-responsive. And we have a lot of confidence in that call because of the processes that we use to get there. And so there, I do think defensibility and validation are very important and should be considered as far as documentation and the approach that you go. That said, I personally don't think that there's anything unique or special with Gen AI that calls for any unique or special validation protocols to be applied to it.

In other words. the same validation that we've been using with respect to TAR, namely whether it's control sets. or validation sampling, elusion sampling, basically using, and again, I just got into the weeds a little bit, but let's step back, to use statistical sampling to show that Gen AI did what it was supposed to do. In other words, sampling on your responsive set, but more importantly, sampling on the set that Gen AI is saying is not responsive. And based off that sampling, being able to show statistically that your use of the Gen AI kind of satisfied the. reason on a standard under the rules and was a reasonable approach to use in your matter, and can be reliable and relied upon. Thanks, Arlan. But again, there's nothing special about Gen AI in my view in that regard.

But again, you have to differentiate it, though, from the internal uses. And by internal uses, I mean things like, well, use Gen AI for issue coding or use it for key document identification or use it to identify privileged documents or to help build your case, right? To find the evidence, to interrogate the data. You have various Gen AI chatbots that are being marketed at the moment. Like all those are internal facing. And for those, I rarely engage in any kind of statistically significant validation efforts. There's lots of times where I engage in no validation efforts for those. I just kind of look at the results and then... see whether it was kind of good enough, whether it worked. And that's a very qualitative way of.

going about it than a quantitative way. But I don't think we need to build in a lot of. quantitative metrics or steps into your internal uses because that increased cost, it slows the process down. There's lots of times where we're just kind of running multiple prompts and just seeing which of those work and for lack of a better way to describe it. And so you kind of got to think about it from those two perspectives when you're leveraging Gen AI, but also other search methodologies like TAR or search terms. So building off of that, then at.

Yes, with respect to prompts that using for kind of the outward facing to, for example, to identify responsive documents and connection with requesting party discovery. The prompt is clearly work product, and it's not even a close question.

Because it reflects the thoughts and mental impressions of counsel, including me, when I'm doing it, about the case, about both the claims and defenses of the case. It's really just classic opinion work product, right? It's similar to an interview memo of a likely trial witness. And the interview memo not only says what the witness kind of relayed, but gives the. thoughts and impressions about the strengths of the witness's testimony, how they appear as a witness, whether they. Make a good witness on the stand, all of that. It's very similar to that. To me, it's a classic work product. And I would say, my only thing to say for the plaintiffs part here is be careful what you wish for, because Gen AI is, I think, materially better for plaintiffs, for the.

use of Gen AI, in many cases, if not most cases. In other words, they're likely to get more responsive documents and less false positives in the productions, potentially faster. And if they are trying to litigate for the prompts, I think all we need is one or two judges to rule that the prompt is discoverable, and you're going to significantly slow down and retard the progress and the use of Gen AI in discovery.

That's a great question. So in my view, the technology has advanced to the point where it is safe to use for logging, full stop. We've been working on this problem, namely the use of Gen AI to create privilege log entries for about two and a half years. And in that time, we've seen the really rapid progress of the technology to this problem. And it's a hard problem because not only do you need to describe the document without hallucinations, for example, but you need to, under the rules, provide enough information about why the document is privilege or the redacted content is privilege without revealing the underlying privilege advice. So it's a very difficult problem. And it's a problem with, frankly, a lot of humans, human attorney'ss struggle with.

When creating privilege log entries. But as I said, the technology has come a really long way. And in my view, it is ready to go right now to use in Privilege logs and to create Privilege log entries. That doesn't, I don't mean to say that necessarily it's a magic button. Typically, there is a fair amount of iteration that occurs on the dataset and the Priv dataset in particular, running it through and refining kind of the inputs and prompts and the like that go into it to make sure that the tool is appropriately creating log entries for that. In that case. Um, but it does get you there now, as far as weaknesses, it's also the weaknesses have been getting less, um, you know, the main one is that the tool, the vast majority of the tools look at the four corners of the document and aren't looking at the family relationship.

Again, there's lots of providers that are trying and working to solve that issue. Um, uh, and so that, and, and it tends to struggle with work product compilations because for the same reason, like a work product compilation, like a deposition prep binder, like the individual documents look like ordinary, of course, business documents, right? It's just the, the collection of them. And perhaps the parent, uh, document that makes clear that this is actually a debt prep binder and therefore, um, appropriate for protection. But these are kind of edge cases, right, and can be solved by other methodologies and workflows, but it can get and can solve and accurately describe the vast majority, if not virtually, most of the privilege documents at issue. Now, that doesn't mean like most people don't, you still want like some sanity checking, and we talked about validation.

Those are helpful, but it's really gotten to be an effective technology.

That's a good question. I think the answer is no to that last question. So we are using Gen AI for HSR responses. It's very well suited for that to try and find the most relevant documents to the transaction, kind of the business course materials related to the deal. It's also being used for privilege logging, typically. Even in the loop methodologies. Uh, this isn't a merger matter, but we, in our case, we had submitted a Gen AI review protocol to one of the government agencies and just waiting to hear back from them. Um, but we're, we're affirmatively proposing the use and disclosing the use. Uh, it's a, it's a challenge right now. I think, um, it's just getting over the initial hump because in most deals you want to move fast.

And the easiest way to move fast in a merger review is to propose TAR 1.0 as your methodology, because that is, uh, that has the least friction typically, uh, I, I think most parties assume that a Gen AI protocol would involve more discussion with the agencies, more questions, more uncertainty. And so, uh, that raises. Kind of a question about whether to use it versus kind of tried and true strategies. So I think it's just getting kind of over that initial hump. If we had, you know, for example, like a white paper or something from the DOJ or FTC encouraging the use of JNI for responsiveness reviews, I think that would go a long way. But, you know, I think parties should definitely feel empowered to use it in merger reviews.

I think it is effective and efficient if it was used.

Right. So we have experience in certain deals with the agencies accepting Cal. But it is not the preferred methodology. I will say I'm a little bit. on an island here, me and maybe one or two other people in the sense that in the context of merger reviews, I prefer 1.0 compared to 2.0. Frankly, I generally prefer 1.0 in most matters because. the techniques that we use, we achieve much higher precision in a 1.0 approach as opposed to a continuous active learning approach. And, you know, it's one of these things that 10 years ago or eight years ago, you know, Cal was particularly useful because it accelerated your time to a given level of recall. That is less of a consideration now that the technology has matured.

Typically, my clients most prefer efficiency, and efficiency is measured by precision, most of them. So anyway, it's a little bit of a tangent, but you are right. We may, as far as large-scale adoption of what tools are used in the merger process, we have the potential to just kind of skip over 2.0 being broadly used and go right to Gen AI.

Yeah, so for me, I think it really ties into what I was talking about, that we typically, if we're going to be using it for responsiveness review, do... enter into at least some kind of protocol or disclose a description of our process to the other side. And that, at least in our experience, tends to limit the disputes down the line because you're describing what the process will be before you're actually getting started with. the process in substance. And it allows the other side to comment. That said, under Sedona and long-lasting principles, we as the producing party are best situated to, kind of design the most appropriate process for producing documents in response to the request. So, we don't kind of think of the protocol as a way to the other side to necessarily dive into our process.

And we typically do not allow that. But there's a lot of information that can be. disclosed to have the other side be comfortable, including, for example, for, the provision of metrics and other data that could be provided for them to show that the.

Right.

It's an incredibly difficult. Problem, and I'm. Unfortunately, the problem is only going to get worse and harder as the technology continues to improve. I mean, we're now seeing if you're on social media, you can see how the how the gen AI video is getting better. Almost in real time, you can see the progress of the technology where a year or two ago, it was very clunky and very obviously fake and kind of used more from a joking perspective. But now, it's getting increasingly harder to tell what's real versus what's not. And, you know, I think there's gonna be a lot of corporate productions where it's not going to be a real issue, because those productions are backward looking, and you're collecting from kind of organic, ordinary course files and sources.

And, So that the issue of deep fakes won't kind of enter into the calculus, but there's going to be lots of cases where it's going to be used and it's going to be front and center. You know, I would expect. So with respect to social media broadly, you saw it was really kind of the family law courts taking the lead on the admissibility of social media and the authenticity of it. But I would expect the same to be the case here where you're going to see it's kind of like your divorces, child custody. um where there might be very high incentives to create a deep fix also employment law context i. think you're going to see that um but you know who knows trade secret matters and the like i do think you're going to start to see more and more of this kind of seep in and kind of the you know what it means for the judicial system and frankly the administration of justice more broadly i think is it's going to be a very difficult problem and let me just add a quick follow-up.

for the rules process to address, right, and the rules committee, given kind of the structure of that process does take years, typically. Right now, we have before the rules committee, a proposed rule 707, relating to AI, although that it's not kind of, It's not addressing deep fakes head on. It's more about compilations and the like being used for trial purposes, but it's not necessarily getting at the underlying authenticity issues. But even that is controversial, right? And so it's hard to like come up with language that can really address deep fakes and to do it in real time as the technology is evolving on us in real time.

Certainly, that was kind of fit, as you described it, fit well with my thinking as well. And part of the reason why I left Big Law to come to Redgrave, you know, it's really the kind of the preeminent eDiscovery boutique in the country. It had nothing really to do with my old firm. I spent my whole career at my old firm. I have many friends there and fond memories, but it really was kind of looking ahead and thinking about the future of litigation, white collar matters, regulatory responses in the context of increasing complexity with data, as well as technology. And the evolution of technologies to meet those complexities, whether it's companies using Gen AI, right, in their own business, and then that's creating. What is essentially new evidence and then potentially using technologies like Gen AI to digest them and to assess relevance and privilege and production criteria and the like.

And that's just scratching the surface. Right. And and also my practice had just evolved over time where I wasn't really working on discovery matters for where my previous firm was trial counsel or white collar counsel. Instead, I was working as discovery counsel for other firms when the other firms were serving as trial counsel or white collar counsel. And that's all that Redgrave does. Right. Redgrave is not trial counsel. Redgrave is not white collar counsel. They are discovery counsel or discovery counsel. And so they fit and can really help specialize in that area. And then what it what it allows for clients and the services to clients that allow the expertise. at all aspects of the spectrum of a matter, right? So you have your trial counsel that you want, but sometimes trial counsel doesn't have necessarily an expertise in eDiscovery or don't specialize in eDiscovery.

We do, and so it allows us to partner with that counsel we call merits counsel so that we can be leveraging our focus on kind of all aspects of the EDRM, and that allows trial counsel or white-collar counsel to kind of focus on the substantive merits. And I do really see that continuing to evolve and an increasing demand as clients kind. of recognize the need and the value, because everything just gets harder. Like our industry and our world, it gets harder every year, and it just is becoming increasingly difficult for the, generalist. third year associate at a big firm to really understand all the complexities of modern discovery, right? And also to be able to deal with it efficiently and effectively because there's a lot of techniques that may have worked fine 10, 15 years ago that now just you would be.

really going about things in an inefficient way if you were still leveraging those.

It's a really interesting question and something I actually spoke to some law students about recently where I came to a law school to give a presentation on Gen AI in the law. And what I said is that the graduating 3Ls are just going to be starting to, their law firms are going to be in the process of leveraging Gen AI, adopting tools, whether it's RV or other internal tools, but also starting to see it used in the cases more broadly. And I think that's really the first class that you can really say that about. And what I told them, and it's not just me who said this, but other people have said this, that it's not that. Gen AI is going to replace lawyers, but lawyers who use Gen AI are going to replace lawyers who don't, right?

And so that, to me, is the most important thing, that one, that the three L's really should focus on leveraging Gen AI in their day-to-day practice at wherever it is, you know, law firms, big laws, small law firms, you know, prosecutors, meeting offices, however it is, that they should be really focusing on that and doing it effectively and really figuring out, you know, all the different ways that Gen AI could help them and make them more efficient and also make their work product better. And I think that is the biggest. And I think the number, the second is really related to the first, but really be open to technology and be open to change and kind of embrace it, because that's what we're going to have over the next several years is just some really starting to see fundamental changes in how we go about this and really be open to that part.

And then the final thing is the exact opposite, which is all the technologies are going to be very important, increasingly important. But if you're going into a law firm, if you're going in-house, if you're going into an organization, you really need to still focus on the human relationships. And because it's the human relationships that will allow you to grow, find a good mentor, find multiple good mentors, right? It's really that connection. All right. is irreplaceable and cannot be replaced by Gen AI. And so don't let that get lost in all the talk.

That it's not magic. It's math, right? It's very complicated math, but it's still math. And there's still a lot to this about the technology and generic in particular, that is still more of an art than a science, like how to leverage it and how to leverage it efficiently. And we see that, like, if you just take your review protocol, and you paste it in as a prompt, your results aren't going to be that good, right? That you really need to refine it and kind of step back and think about, well, okay, how could you, how can you write this in a way to make the gen AI be as successful as it as it can be and analyzing the data set? And that you, gen AI can do a lot of things, but it's not perfect.

And maybe this gets back to how we started the conversation. At the end of the day, you're still need and have to leverage and have to grow your own human expertise, human experience, and use that expertise and experience on your case. And it's like a supercharged tool, but it's still just a tool. And it responds differently to how good the person is with the tool. And so it really rewards kind of an investment in time, but also like taking the results and you still have to build the case. You have to figure out, well, which of the docs that it's turning back from you are going to be a good deposition exhibit or a good summary judgment exhibit or what could play well before a jury and like all of that.

But it's still irreplaceable.

Well, thanks again, guys. I appreciate you having me.