Judge Xavier Rodriguez on AI, Proportionality and the Future of Discovery

Well, thank you for the invite. And it's always good to be at NeLI.

Yeah, it's amazing to think that 10 years have gone by now since those 2015 rules and, you know, the dreamed of goal of reaching proportionality and cooperation, I think, just as you allude to, has been overwhelmed by just the amount of data and the differing data sources. You know, back in the old Zubelik days, when Judge Shinlin had those in 2001, 2002, we thought emails were the bane of our existence. And, oh, my God, that just seems like such a quaint time. But, yeah, the rule, the goal has not been able to keep up with collaborative devices. How do we get stuff out of teams? Mobile phones, ephemeral messaging, it's all just. increased in complexity, and yeah, we're still fighting 10 years later.

Yeah. So a number of things there. Let me see if I can break it all down because we'll be talking about. AI use and eDiscovery and what kind of fights we can anticipate there. And then we talk about AI use in terms of legal research and presentment of briefs and motions to the court. That's sort of a separate issue in my mind. The third issue in my mind is when we talk about AI usage, I mean, the clients, as you said, are already using AI. I mean, just to give an example, in the employment area where I used to practice a lot, 25 years. ago, 30, whatever it's been, you know. A lot of employers are using AI tools to assist with the hiring process, screening process, and even the RIF process, reduction of forces at the end.

And so, you know, let me pick on my colleagues, the judicial colleagues. You know, the idea that we're going to make this go away is just not going to happen. You know, as much as some people have reluctance to all this, you know, we're going to have to adopt a responsible use mentality. I understand the concerns about AI. I have some too, right? I'm concerned about the weaponization of AI in the military. You know, we definitely want humans in the loop there. I'm concerned about, you know, what kind of employment impacts there may be. I'm worried about, you know, are we all going to become dumb, and what are we going to do with college kids in K-12, and what's the impact? going to be there?

There's a lot of good, valid reasons why we need to be cautious. But that said, I'm taking the attitude in my work, hey, there's some positives here in trying to get to just speedy and inexpensive. How can we responsibly adopt some of these AI tools to get to a quicker resolution for our clients with greater accuracy? So with that being the background, let's talk about these three different buckets that I've talked about. AI and eDiscovery, right? So the goal that I think a lot of. eDiscovery vendors or might be hawking out there, maybe a little too prematurely, but maybe they're going to get there, is that we're going to replace first review with AI tools. And we'll just have humans now doing the quality review afterwards.

You know, that's a noble goal. I hope we get there. I'm not sure we're there yet, but hopefully we're going to get there. We're going to have to have humans in the loop. But in terms of responsible AI usage, the first issue we're going to be grappling with is, well, how are we going to validate the results that the AI tool identified as responsive documents? And so we're going to need some validation there. We all know on the second bucket that I talked about, we've heard about the countless hallucinations that have continued to take place. I go around telling everybody, it's not rocket science, folks. You use an AI tool. All you do is you check whatever citations the AI tool has. given you. You, one, make sure that those are real cases or law review articles, whatever that case may be.

You check the site to make sure it actually exists, that it's real. And then you do the second phase that, you know, whatever the AI tool gave as a response that citation was. crediting, if it was a quote or an underlying principle, you may not be able to do that. You may not be able to do that. You may not be able to do that. You make sure that case, you know, stands for that proposition. but it's not rocket science, right? And we keep on focusing on all the hallucinations. Someone's keeping a database of these cases. And it's like, I find that kind of odd. You know, if we're gonna keep a database of this, well, then we need to have some more information in this database.

What tool was being used that made this hallucination? Because maybe we need to know which tools are doing a better job than other tools, but those databases aren't doing that. The other thing about these databases is. no one's keeping track of how many times a lawyer has responsibly used one of these AI tools and it is correctly identified the cases or thatutes and it worked as intended. So, you know, the that there's 200 cases, without having an understanding of what else is out there that worked correctly is kind of meaningless to me. But boy, my colleagues are, way overreacting on this, but I would posit to say that a lot of the lawyers who had made these mistakes about turning in these briefs probably are the same set of lawyers who did it old school.

They had their paralegal do something, or a law student, and they just signed off on it, and no one was claiming that the world was coming to an end back then. And the third bucket, and maywe we'll have time to talk about it today, the third bucket is going to we the most interesting for me, is clients are using these AI tools, and so. then how are we going to get these into evidence, and what are going to we the evidentiary objections, and whether or not the current proposals are going to get us there or not get us there, that's definitely a work in progress.

So, a couple of links there. So, Judge Starr rightlyfully acknowledged from the very weginning that, hey, this is a new tool and sort of emphasized, if memory serves me rightly, about, you know, he sort of emphasized the human in the loop aspect of the process and sort of in very empathetic terms about the humanity of the practice of law. And what we do, and that these were uncaring tools. I'm paraphrasing, but that's how I sort of rememwer his order. You know, we saw a backtracking of not only Judge Starr's language, but other judges who also either prohibited or wanted disclosures, wecause it's wecome quite apparent now that's unworkable. I mean, do we as judges want somebody… I don't know. we disclosing every time they used Westlaw Co-Counsel or Lex's AI?

I mean, I certainly don't. And, you know, and if they disclose that to me, well, so what? I mean, how does that change. whatever I'm doing? So now I will say this, and in light of the continued failure to check work, I, and it's kind of funny you mentioned this, I'm sharing rightly now our local rules, rules committee, and we're revising our rules. I certainly have not run this, and this is not a vote of my district, but I am contemplating inserting language in there, just reminding everyone that they had to check their output. And, and I'm gearing that more, not so much to lawyers, but something that we can warn pro se litigants about, wecause I think that's the more troubling part rightly now.

Yeah, so… I mean, this is my own personal thoughts, rightly? So I don't want anybody thinking what I'm saying is going to we what some ethics commission that you might we in front of is thinking. But so my personal thought is, you know, it's probably a smart idea to let your clients know, hey, we use AI tools for the following purposes. I mean, I'm just thinking practical, rightly? I've got a client. I want to keep that client. I want that client returning to me. And so I don't want any mess ups or unknowing use that, you know, they may not like it. So I would just we transparent with the client from the get go. And I would certainly get permission from the client if I was going to we using even enterprise AI tools to upload any kind.

Sensitive information. I mean, if I was a patent lawyer, I'd we getting permission from my client, even on an enterprise AI that I'm. going to upload source code or whatever the patent technology at issue is. And so that's just, I mean, I'm just thinking realistic attorney client relationships. There's ween some thates that have issued ethics opinions, my own thate as well, Texas. And of course, it gave the ever helpful, it depends about whether or not you have to disclose. I wish someone would just say you have to and under the following circumstances, just so we all have clarity. So along those lines.

Yeah, the reason I'm smiling is I just had an interesting debate a couple of days ago with two lawyers about the transparency of the use of AI in eDiscovery production, and is this going to we the TAR wars all over again? And so I'm, you know… In many ways, I'm just rule based. And I would say that, hey, until such time as there's ween a deficient production, all these judges who have ween doing all this, I call it mother, may I or dad, can I, you know, can I use TAR? Can I use AI? Hey, here's our electronically stored information protocol. And, but we got some gaps in it. So judge, fill in the gaps for us. I mean, all that from a rule based standpoint, I think is completely wrong.

I mean, the way our, rightlyly or wrongly, the way our civil procedure rules are written is that a requesting party request, a producing party produces, and then under the federal rules, under Rule 26, you know, If a deficiency can we shown, you just can't say, hey, there must we more here. Your document production was insufficient. But once you have shown a deficiency, then you can file a motion to compel, and then it gets teed up to a judge. But that's not what's working rightly now. That's not the world we live in. The world we live in is there's a lot of judges, primarily magistrate judges, and I'm not faulting them. They think their job is to move the case and keep the scheduling order in place and move everybody to a trial setting.

And so in their mind, rule one, which requires us to engage in just speedy and inexpensive determinations. And so in their mind, rule one, which requires us to engage in just speedy and inexpensive determinations. discovery disputes at this pre-production stage. And, you know, for the, for the, for what I'm going to call them, TAR II, TAR III, whatever, whatever series of TAR wars we're going to be in now, it's going to lead to a lot more questions. I think it's going to be even more difficult than the TAR wars. And so the level of transparency, you know, at least. I mean, you want to disclose that, hey, I'm using AI. And if so, then what tool am I using? I don't see what the harm is on that.

If you're using it just for early case assessment and developing what keywords you will use, if you're going to use the old TAR II, but you're using AI for just that development of keyword searches, I mean, I really don't see the harm in just disclosing that. Now, do we want to get into the actual keywords that tool gave or not? I mean, now we're getting, I'm going into a spectrum of where some lawyers will think, well, this is all fair game for transparency. And other counsel will think, you know, I don't have to give that up. I won't give it up. But all I'm suggesting to folks when I'm speaking on this subject is. I think these AI tools give us a lot of potential, and I'd hate for that potential to be killed prematurely, just like I think we didn't completely kill TAR, but we came awfully damn close.

You know, and I'd hate for the same, I think it's a mistake. I'd hate for the same mistake to happen that litigants, the clients will not want to use these tools and not have their outside lawyers use these tools because they don't want to go into the discovery fights or even the pre-production fights in the elongated sessions to negotiate ESI protocols. And they're just going to say the heck with it, don't use it. I think that would be a grave mistake.

Yeah. So how to tackle that one. Okay. How to show deficiency? You know, generally you can show deficiencies because you took a deposition and the deponent said, oh, no, we had these email exchanges or we had these chats. And all of a sudden during the production, no emails or chats were produced. I mean, it's obvious now there you've been able to demonstrate a deficiency in the production. Now, going back to proportionality factors and all that, you know, is it still necessary? Were those additional materials redundant or were they unique? You know, we have all those kind of challenges. But in the use of an AI eDiscovery tool, you know, I think we're going to have to have validation, just like we had TAR and we had validation there.

I think we're going to have to have validation. I've been recommending to the eDiscovery vendor community, hey, if you're going to build out these. tools, built into these tools, some kind of validation component to it. So you can show a judge, hey, we hit 75% or we hit 80. And so both the opposing counsel and then necessary, if you have to show it to a court, we got some kind of objective number there. But the complexity I see here in the AI is, you know, how, if we have to go to these fights, I mean, you're using an AI. discovery tool, you know, you can perhaps shed some light about, what prompts you might have used if there were non-attorney work product privilege kind of information.

We're going to get into the fight over the, and let me backtrack. The way that most of these tools, but not all these tools, have developed is that you train the tool a little bit. And so you will put into the tool what used to be, I think, attorney client, pardon me, attorney work product privilege material. Back in the old days, and this is showing how old I am, you know, They would get a whole bunch of us young associates and say, hey, we've got this major case coming up. We need you to go to the warehouses. Here's what the case is about. Complaint, complaint. Here's the request for productions we got. Copy, copy. Here is now our internal document where we explain what we think are all the key factors of the case, the key issues of the case, the key players, the relevant dates, blah.

Hand all that out. Yeah. And then we would all individually study those documents and then go to the warehouses the next day and flag all the relevant documents we thought were relevant. You know, any number of mistakes could be made in that process. Right. I heard it one way. Brandon heard it another way. Daniel heard it another way. And and then fatigue of just going through warehouse boxes. But that was the process. Now we go here to an AI world and is that sheet I described that was going up and training and served as a prompt for the training of the AI tool, is that going to be a training work product? I'm sure we're going to see fights over that. Now, not everything is a training work product, right?

It's just talking about facts. That's not going to be a training work product, but we're going to have those fights. We're going to probably have fights over what the recall rates were out of these discovery tools. And arguments can be made on both sides that, oh, even though it was a low recall rate, that didn't mean anything. But then how do you explain why it didn't mean anything? And that's going to be a good little fight. The validation studies, hopefully we'll get the vendors on board for that. I think we're going to have a fight over. If not part of the validation study and independent sampling of the non-responsive documents, it's, uh, I think we're going to still have to have that old tried and true method.

You go back, see what was not produced and do a sample set there, uh, to ensure that you, your, your prompts were correct and the output was sufficient. And so, uh, yeah, uh, I make it sound so easy. I don't think it's going to be easy at all.

Yeah. So first things first, right? You got to hit relevance. And, you know, so at the very beginning, I do my own Rule 16 conferences, rather than give them to the magistrate judge to conduct. And I'm not surprising anybody. I put out in the order setting the Rule 16 conferences, what questions I'm likely to ask you. And I ask you to lay out what the causes of action are in the case and the elements. Now, why do I do that? People are shocked sometimes when I ask those questions, because they're like, what is this? And then more shocking to me, though, is sometimes people don't know what I mean by elements. And the reason I ask those questions is when we start talking about their discovery needs, I ask, guys, how is this request relevant to any of the elements and the cause of actions that are being alleged here or any of the defenses?

And then I'll get this pause. I'll go, oh, you're right. So, you know, relevance is the first one. And then for the proportionality factors, you know, when that rule really got elevated in 2015, people started doing these, judges started doing first proportionality factor analysis, second proportionality analysis factor. I mean, it's like. I'm not going through all six and writing 15 or 20 pages on some discovery fight. Just isn't going to happen. And so I will have you in a hearing, and I'm just going to ask, okay, what have you already got? You're asking for this. Why does this advance the ball any more than what you already got? And I'm looking for sort of marginal utility. Why do you need this? If you already have all this, then what?

Why are we fighting over this? And you need to have an adequate explanation for me for that question.

Yeah. So let's break it apart. I believe in discovery plans, right? I want to know how many depositions you want to take and why this many, but that's different in my mind from the ESI plan and protocols. I don't believe in those actually. ESI protocols, I've heard some real good reasons for why we should have them. And when I talk to like paralegals or other. staffers in-house counsel departments or corporations, they want these ESI protocols because they want to know what am I searching for? And we want to do, one and done and that's it. And I can appreciate that perspective, but that's not actually the way discovery is done. It's not one and done. Right. And and so that's my first problem with these ESI protocol orders.

The second one I have is generally they're doing these things and entering into these orders and asking judges to sign these orders before they ever tested them. And so some of these ESI protocols, you know, once it actually gets into the production phase, people find out, oh, my goodness, this is way over inclusive. Or, oh, my goodness, this is way under inclusive. And then now we've got to go up to the judge and try to explain why this is wrong and why you have to assert a good reason for the change in the order. And I'm like, why did we enter in this thing in the first place? You know, and it goes back to producing party produces, responsive party, producing, requesting party requests, producing party produces.

And until such time as the deficiency shown, then why was a judge involved in this whole ESI order business in the first place? So I don't necessarily think there's a lot of value in these ESI orders. And maybe I'm. Unduly influenced. Because by the time it gets to us, right, we only see the failures, right? One party thinks, oh, we got them now. They agreed to this ESI protocol. And so now we're going to make them stick it to them. And judge, don't let them get out of it. And I'm always trying to, like most judges, I think, trying to get to the merits of the case in a reasonably efficient manner. And so I have a difficulty with that kind of an attitude. And then I have a difficulty, too, with the counter argument I sometimes hear.

Oh, judge, well, you know, so they're saying this is a hot document. Well, that may or may not be true, but that we did not produce. Yes. But this ESI protocol that we all agreed to, that wasn't required to be produced. I've never understood the rules of civil procedure to allow a party not to produce a relevant piece of information that it knew to be relevant, and now you get to hide it under the auspices of some ESI protocol order. I don't think that's the way the game works.

Yeah. So, you know, The predicament you talked about could probably still apply to the top 40, 50 schools in the country, right? You know, yeah, you're taught to think, you're taught to look at different perspectives. presented an array of courses, you know, on some topics that you'll never touch upon again in your whole career. And so then you step foot in a law office and they ask you to prepare a TRO and you. panic because you have no idea what the hell you're supposed to do. So are we doing a better job today? It's a work in progress. There's a number of schools now, it's not in every school by any means, far from, but there's a group of us, maybe 30, 40 of us, that teach eDiscovery, primarily we're adjuncts, but there are some full-time faculty members who teach eDiscovery.

Bill Hamilton from the University of Florida has done a good job of sort of shepherding all of us. and there's a platform that we all share our syllabuses and thoughts so we can better prepare our students here. And so that's been very helpful. I think we need to take that much further. than just the 20 or 30 of us that are part of that little group. I think a number of places now are popping up with sort of emerging technologies in the law kind of programs. Stanford, Suffolk, Northwestern, the major ones that I'm thinking about at the. moment off the top of my head, but there's got to be a lot more. I started teaching a course last semester that I called Emerging Technologies in the Law.

And it was an overview of AI, cybersecurity matters, cryptocurrency. And so it was an introduction to the class about just where I see that in all likelihood in their practice, they're going to encounter some elements of these things. I mean, even family lawyers now are seeing cryptocurrency disputes, you know, someone hides the assets and puts them in crypto. And so you have to have a basic understanding of all these things. We definitely have to up our, game about that uh yeah marketing that's uh so marketing uh the economics of a law firm uh that is something we definitely have to hit uh because a lot of our students are going to open up their own shingle or pare up with someone uh and so one they barely know how to.

practice law and they've never opened a business and now we're asking them to do both and so we definitely have to help them out there uh yeah that's there's so much to.

People. So I'm lucky enough to get a lot of invitations. And so next week, I'm heading to Georgetown. And so I'm very fortunate to be on the on the speaking circuit. I do it, one, for me to learn. I learn from my fellow speakers. I learn from the questions that are being asked of me and the other panelists. And so one, for me, it's a learning opportunity. Frankly, I think I learned more than I share with everybody else. But then number two, just people. And so I've been real fortunate along the way on all this. I'm not even sure how I got on the speaking circuit to begin with, but I was, I always joke around. I was the second tier, you know, Judge Shira Lynn, Judge Peck, Judge Francis, all the icons of eDiscovery were the first tier.

And as they all slowly began to retire, somehow I got elevated. And so, but along the way, I've met some incredible practitioners, incredible judges, and we've developed long lasting friendships as a result.

Thanks for the invite, guys.