Judge Andrew Peck on TAR, Generative AI and Whether the Rules Need to Change

Happy to do so. Good to see you both.

I mean, it is a great conference. And it is perhaps a little smaller than some of the other eDiscovery conferences. But particularly, for those in what New Yorkers might call flyover country, it is the biggest and best conference in what I'll call the Midwest, whether that's a correct description of Kansas or not. It is big enough that it gets at least. five or six, sometimes more judges participating. The faculty is the same, you know, A-list faculty that one would see at Legal Week or Georgetown Advanced eDiscovery Institute. But it's small enough and Robin has added a faculty and all participant dinner after the first night of the program. So really the participants can get to meet everybody and talk to their hearts content, not only in the formal sessions, but in the other sessions.

How I got there is I met Robin and John Sablone, who was helping her in the beginning at any discovery conference in Boston and either at that time or shortly thereafter, Robin called their email and said, hey, we wanna do one in Kansas City, are you willing? And I said, sure. So sort of the rest is history. The one footnote, it is true, I've been to all 10, but in full disclosure, for two of them, the schedule conflicted once with the Yankee playoffs and once with the New York Rangers hockey Stanley Cup playoffs. And so I didn't wanna travel since I had tickets. And so I said to Robin, I would video in for my panels and to listen to the other panels. You know, I am always willing to talk to you discovery, but Yankees and hockey Rangers are my first love.

Well, a person has to have priorities. That's a lifelong fan. Sometimes those things get in the way. Of course, you know, Robin has brought me twice, I know three times now at NeLI to Royals Games and even gave me a present. And then of course insisted I wear it to the last year's Royals Games. So I was sitting there with my Yankee hat on, of course, and my Yankee shirt was then covered by a Royals T-shirt.

Well, you know.

Well, I think, you know, the industry embraced my decisions in terms of the use of TAR. It didn't embrace it in terms of, I would have expected, you know, particularly after Rio Tinto, the second of my TAR trilogy, I would have expected there to be much more use of TAR. It's still, you know, a keyword search industry with TAR layered, you know, as a replacement or on top of that. And it shocked me that the uptake or uptake of TAR was much slower and remained slower.

I'm not sure it would have made any difference. The problem is lawyers still are a very conservative bunch in terms of use of technology or things like that. Perhaps in Rio Tinto, where the issue of validation came up, the later in the case, I might have said something differently. Instead, I guess I, like many of the lawyers, didn't really want to do heavy math. And so when they were fighting about doing validation there, I suggested several times to them that they employ more grossmen as a special master to deal with them on that. And they were hesitant at first and finally they agreed to do it. And I was very relieved and Maura took it from there. And indeed, her career has changed considerably as a result of that special master experience.

So it was good for both of us.

You know, I think first of all, we have to remember that there is the bubble, the Sedona conference bubble, NeLI attendee bubble, et cetera, of people who want to learn about TAR and other things. And then there is the rest of the bar that barely knows how to produce emails, let alone any other newer technology. And for the cases, most cases, it's fine. It is the big cases, the MDL cases, the class actions, the major suits against Fortune 50 companies or whatever that need it. The typical employment discrimination case, even though the smoking gun documents may well be in the ESI, the typical auto accident, although there may be things in the so-called black box in the car that if people thought about it and tried to get into that it might well determine the cause of the accident or anything else.

But most of those cases, they don't think about eDiscovery, let alone use of TAR. And obviously, I mean, it takes a certain, it's a much smaller number than I think many lawyers realize, but you have to have a certain amount of data for TAR to be worthwhile. The other factor that I think still exists is the TAR tax concern. And what I mean by that is if you're doing keywords and you've agreed with the other side on the keywords, however bad the associates or contract attorneys or whoever doing the review is, it sort of is what it is. If you're using TAR, then the other side may go into court and insist on certain validation protocols or insist that they get a say in how you are doing the TAR process.

That cost of going to court may well take the cost savings from using TAR and exceed that. So some lawyers would say, I'd rather use the not as good quality old fashioned way of doing it because the judges are familiar with it, then do a better job using TAR, but then have to have motion practice to explain it all. You mentioned before that some lawyers don't even know how to produce emails. And it's interesting.

You know, some of it may come about when judges who are now, I think, more sophisticated than the bar can turn to lawyers and say, why aren't you doing X or it would save your client money if you did Y or the less of a carrot and more of a stick, you know, you are putting eyes on every document that hit on your search terms. When you get to the end of the case and assuming you're the prevailing party and you asked me to award you attorneys fees, I'm not going to award fees for conduct that could have been avoided by using tools, whether that tool is TAR, whether that tool is threading or deduplication to cut down what needs to be reviewed. So and there is a little more teaching in law school about the eDiscovery process.

So we're getting there. You know, I'm amused by certain things. You know, in 2006 or thereabouts, when the first of the federal rule changes dealing with the SI came into experience, you know, lawyers would be before me and they would point to their gray hair or lack of hair and say, Judge, I'll give them every paper memo that the client has. I'll give them all the letters we wrote, but please don't make me deal with email. I can't handle that. That conversation is repeating nowadays somewhat. I am told by my former judicial colleagues, but now the lawyers are saying, I'll give them all the emails that hit on the search terms or whatever. Emails, no problem, but don't make me look at text messages or Slack messages or any of that.

So the more things change, the more they stay the same or whatever.

I mean, I think the problem would be harder in criminal cases, but the first time you say X method of communication is not discoverable. That's where all the smoking gun documents will go. There's no doubt about that. So it is really up to the lawyers, the clients, and if brought before the judiciary to deal with it, both by education and by use of proportionality, but it has to be educated proportionality. There was a case that I covered six months ago, maybe on the EDRM case law podcast we do monthly. And it was one where the plaintiff in the case was a corporate entity. The defendant was a plaintiff, which it was a individual sort of a flip of the more common case. And so plaintiff corporate counsel signed a stipulation saying neither side has to produce text messages and things like that.

We will only use emails for discovery. And after the first deposition learned. that the defendant individual didn't really use email. They use text messages and the like. And so then they came back to, this was in New York State Court, the judge and said, well, we need this stuff. And the judge basically said, terribly sorry, you agreed not to do that. I'm reading between the lines. I think that the corporate party thought it would be too difficult for them and didn't need it and et cetera, et cetera, but they were wrong. We're seeing it now in the hyperlink cases that are coming down. In one of them, I think it was the first of the two StubHub cases, but don't hold me to that. The parties agreed StubHub stipulated with the other side that they would produce the hyperlink documents.

And then lo and behold, figured out that they couldn't do it. And the cases went in front of the judge who basically said, you not only agreed to do this in a stipulation, but you had me so order it. So it's no longer a stipulation. It's now a court order. You should have figured out what you could or couldn't do before you agreed to something. And just to finish that situation, almost a year later, they did what they were allowed to do under that prior stipulation and order, which is ask for the court to change the requirements for good cause shown. And they came in with all sorts of affidavits from their IT folks and outside consultants saying, well, we may have agreed to this, but it's not possible to do it on a big scale like this.

And the court said, okay, you convinced me, you don't have to do it. Or you only have to do it for a very limited number of crucial documents or whatever. So it's always, only agreed to what you can do and make sure before you agree to carve something out that it's not something where the data you need is gonna be.

That's right. I mean, when I was on the court in simpler times, I had the expression, not to insult anybody, but I called it bring your geek to court day. And what I meant by that was if the lawyers are arguing in front of me or the court generally, as to what can or can't be done, it's basically gonna be, he said, she said, situation probably with one or both sides not knowing what they're talking about. So bring in, whether it's the corporate IT person, whether it's an outside consultant, bring in the person who can explain to the court how all of this sort of stuff works. And that really needs to be even move further backstream. And that is before you agree to do something in an ESI protocol, whether it's a stipulation or an order, make sure you have, as the lawyer, talked with inside counsel, talked with the clients, IT people, talked with whoever you need to talk to and not just say, well, in the previous case I did, I stipulated to X and it seemed to work fine.

So I'll do that again. Doesn't work.

Well, first of all, I think it is going to encourage greater use of TAR. Because lawyers won't use the most current best process. And I've said this publicly a few times. So they'll say, all right, I'm not gonna use AI. Oh, TAR, yeah, that's been around enough, I'll use that. So that's good. But then a more serious response to the question, lawyers have to know what they're doing. The biggest thing we've seen with AI are the hallucination cases where lawyers used AI to research and or write a brief to the court. The Mata Against Aviankic case is probably the most famous of it. And that got, you know, not only legal press coverage, but New York Times and other major newspapers. And yet since that, there have been at least half a dozen, probably 10 subsequent cases where lawyers use AI.

Submitted briefs with false citations. So that was a problem. That's a different use of AI than using it for discovery. But, you know, and I guess my concern is, you know, what if I had ruled in the silver more that TAR was no good? So my concern is that the first fight over the use of AI for discovery comes before a judge knowledgeable enough or who becomes knowledgeable enough to deal with it. Because if one gets an early case saying AI is not good for discovery purposes, that'll set use of it back five years. Now, you know, the other factors are cost. At the moment, there is certainly a cost to using AI in discovery. It's coming down as always happens with newer technology. But it, you know, I don't know that even in the big case with companies that can afford to spend a lot of money on discovery that it wouldn't be cost prohibitive to use AI when you're dealing with a corpus of potentially a million documents.

So that's a factor. The ability to do the prompts and to do the prompts carefully, appropriately, is going to be interesting to see how that develops. And then just like there was what I call, I know there's called the TAR tax, there may be an AI tax. Well, if you use TAR, you know, you may have to disclose certain things but you don't have to disclose other. If you're using AI and you've got to disclose what prompts you gave the tool in order to call through the documents, I think a lot of lawyers are gonna be uncomfortable with that. And so that's gonna be another factor. I am, you know, hopeful that if it turns out that AI really can be used well for doc review and, you know, certainly, and I'm not, I guess I'll say probably the biggest vendor in the industry has an AI for review tool.

I won't mention the name to stay agnostic even though I'm no longer a judge. So I don't have to be, but so apparently it can be done. I have not yet used it. So I can't say whether, you know, There's always the salesperson hype versus how things actually work. So clearly it may or may not be there already, but it will be there in a year.

Yes, although I will say this is not a new phenomenon. It's old wine in a new bottle. I mean, before we went to, you know, like constant use of Westlaw and Lexis and all of that when lawyers were still using the, you know, decennial digests and the other tools like that, I could tell by the way the lawyer formatted the discussion of a case that they had just lifted that from one of the print sources. So it's not a new problem, but at least there the case was real. It may or may not have said what they cited it for. Now we've got, you know, made up cases. So but, you know, it goes back to the good lawyers versus bad lawyers. And just, you know, my favorite thing from Mata is when the fact that Avianca's lawyers couldn't find the cases that Mata's lawyers cited.

And they and then the court said, you know, Mata prove that these cases are real and they verified their work. They asked ChatGPT, are these cases real? And ChatGPT said yes. And that's what they told the court. I mean, you can't make this stuff up.

Yeah, I mean, I think so. And part of the difficulty is I understand it is. Using slightly different prompts or indeed, I think even using the same prompt at different times, you can get different results. So, you know, if I were on the bench and this came in front of me, I'm not sure how I would rule on whether that's discoverable or work product or not. It's, you know, we'll see how it all plays out. And, you know, there may be entities that come up and become the prompt engineers for anyone willing to pay their money or whatever. There's a business for everybody. I don't know. Yeah, there is. But sorry, just I just want to circle back on that one thought that you had there.

So is it, do you foresee based on what you said, if I were on the bench now, I don't know how I would rule on this.

Yeah, I think so. And that's always the fear with new technology now with TAR is not so new technology, but we're still seeing fights on its use. You know, to a certain extent, its economics, you know, whatever you save by getting through doc review more quickly with AI, are you going to spend it supporting it and worrying that not you're going to spend the money for the motion, but that the judge might say, no, you got to give the prompts or worse, you got to redo the doc production a different way.

Part of it is what is being produced. I said in Da Silva Moore, I don't think discovery like this is subject to the Daubert test. If we start saying you got to have an expert to apply on everything you're doing in the discovery process, rule one, just speedy and inexpensive will be out the window, at least two of the three of those terms. So, are the documents that one would expect to find? And usually there are communications going across the V so that the plaintiff has some idea what documents the defendant has. And then secondly, and this does, although I was not a big fan of validation protocols, it does require a certain examination of what wasn't produced because you know the old you don't know what you don't know, you may have produced a lot of things that look like you've covered the waterfront and then you discover that the things left behind as non-responsive are very responsive.

So there probably will have to be validation approaches to this. I think, you know, we are getting, we have gotten to the point, I think, where everybody who uses TAR seems to agree that there has to be validation of the TAR process. They may not agree on, you know, how one does that and how much detail, etc., etc., but I think that's going to have to occur for an AI process as well. And fortunately, you know, promo for DLA, we have a whole very advanced AI group at DLA. So I don't have to deal with this. I like that. And you shouldn't have to deal with a judge either.

teams or Slack or text messages or even any other database, there is a lot more than 2006, 2015. Here we are just started 2025. Should we take a look at the rules again? always good to take a look at the rules, but you know, two things. One, as you mentioned, it takes five years for a rule to go from being considered or proposed to the advisory committee rules to actually coming out the other end. It also takes a fair amount of consensus. And you know, going back to 2006, you know, people were upset or surprised that the rule did not specifically refer to email. It referred to electronically stored information and people were saying, you know, why are you not talking about the current way we deal?

And the answer is because it's going to become outdated and the rules are forever. So, you know, I would think, for example, hyperlinks, that's going to get resolved. You know, all it takes is Google and Microsoft to figure out how they're going to make it easier for their customers to gather the hyperlink documents that they need. And we're already seeing some changes there. So I think that's going to get the debate on the hyperlink document production is going to get resolved by, you know, commercial, technical, practical things. AI might justify a rule amendment, but I'm not sure what it would be at this point. I mean, one thing I would say is there are a number of judges who by their chamber rules have prohibited the use of Gen AI or have said, if you're using Gen AI, you've got to disclose it to the court.

And, you know, the problem with things like that is they're badly drafted. I mean, at least one of those rules was if you're using AI, you got to tell the court, well, if I use spell check on my brief, do I have to disclose that? You know, it doesn't, it didn't say if you're using generative AI. So, you know, one has to be careful that the people making rules know what they're doing. I think we need to see a little bit more of what's going to happen with AI for discovery and AI for legal practice before there can be any serious changes to the rules because, you know, if I were to be the czar of rulemaking, what would I say now? Yes, you can use AI if you do it the right way or something.

Does that, you know, that avoids the judge coming out with a Neanderthal opinion that sits things back five years? But I'm not sure it really adds anything. So, you know, and as I say, it has to be something that both sides of the V agree on. Look at what has happened with the proposed rulemaking on privilege logs. There were a gazillion comments to the rules committee in response to their request. And essentially, those who are mostly responding parties, defendants said it's burdensome. We need a faster and better way to do privilege logs. And the plaintiff side came in, no, you know, the defense bar is just over claiming privilege. And the problem isn't the logging. It's the review for privilege. And they need to do that.

And they need to do it better, which led to the compromise of adding to rule 26 F and 16 B that the parties should talk about how they're going to deal with privilege logs. Is that really, you know, in that in way, that's common sense. And secondly, it doesn't give guidance to parties, particularly those who are not as sophisticated as to what the rest of us are all talking about. So, you know, rulemaking is appropriately or inappropriately depending on how you want to look at it, much too slow and comes in late in the game. And we'll just have to see how AI for discovery, you know, comes out before we get into any rulemaking on it, I think.

I mean, I think the younger or newer judges, both magistrate judges who wind up getting most of these discovery disputes and even, you know, the newly appointed district judges are younger, have had more experience, has practicing lawyers in dealing with technology, but the other point is judges are generalists. You know, we have to deal with everything in the federal system from civil discovery disputes to criminal, the criminal docket and back on the civil side to prosa parties, employment discrimination, I think prosa parties and employment discrimination cases are something like 25% of the federal court docket. Those are not the cases where you're going to have these massive discovery fights. And so like, and then, you know, patent and cases that, you know, one has never heard of probably PACA, the Perishable Agricultural Commodities Act, which came before me a number of times, much to my surprise, because I didn't know what it was.

And, you know, it is incumbent in anything for the lawyers with or without their geeks to educate the court. And that remains, whether it's a discovery technology issue, whether it's, you know, the merits of a lawsuit, you know, in the patent world and in the product liability world, they often have what's called a science day with the court, where the parties agree on a non, you know, leaving one's position at the door to bring in either the lead lawyers or and or experts to explain to the judge how a, you know, knee replacement, the product works or how, you know, what a certain drug does. And there's no reason why lawyers can't, and indeed they should, say to the judge, you know, we want to get your approval to use AI to do X, Y or Z.

Can we come in before you on a science day type approach and explain the technology to you? And most judges, if they're too busy, they'll refer it to the magistrate judge being somewhat facetious, but not entirely. You know, the judges want to get it right. They also want lawyers to, you know, not fight over stupid things in discovery, but I would say, you know, whether one can use AI and how is the sort of thing that judges need to apply not.

I would say embrace technology, learn how it can make one's life as a lawyer easier. And, you know, whether it's using AI in discovery or whether it's even the simpler, you know, using AI for legal research, but using it appropriately, using AI to do first drafts of a letter or an email you know, the sum of which is the sort of, I wouldn't say grunt work, but it's the sort of work that if it can be done faster and cheaper for the client by using some sort of technology, you should be doing that and you should not be afraid, even if you don't know how the tool works to ask somebody at your firm to teach you or just, you know, yes, it may not be billable time, but go sit around and play with the tool and figure out how it can make your life easier.

AI is not going to replace lawyers. The articles about that are, you know, overhype, but it is going to say that the way we practice, the way we bill for tasks is going to change and you should be the lawyer who has the skill to use that as opposed to the lawyer who might get replaced because the task their expert at is now something that a machine of one sort or another can do.

Happy to have done so. Great to talk with both of you and I hope I will see many of the podcast audience on September 11th and 12th, 2025 at this year's NeLI.