Magistrate Judge Allison Goddard on AI in Chambers, Prompts and Proportionality

Oh, thank you so much. It's an honor to be here. NeLI is one of the best eDiscovery conferences that I've been part of for several years now. So I love the idea of continuing the learning that happens in that short period of time and stretching out across the air. So thanks for having me be part of this. Well, I love that you said that, by the way. So thank you very much. So this is really kind of breaking news, I want to say. So let's get right into it. Let's talk about the Southern Conference and AI guidance. So the Southern Conference has always been this leader, I'll say, in eDiscovery best practices and the guidance that it puts out and the publications that it has.

And most recently, you teamed up with some others. I want you to talk about please who it is that you teamed up with and talk about this recent focus on AI for the judiciary. And how do you hope that this guidance is going to shape the way that courts and I'll also say litigants approach AI-powered discovery. So let me give you the backstory. I'm going to say it like that, but it's probably not going to be as interesting as that sounds. But let me give you the backstory. The guidelines for the product of an ABA AI working group and then Sedona, the Southern Conference had the leadership of being the first to publish them. So the working group is was headed by Judge Scott Schlegel, who's from Louisiana, who sits on the Louisiana Court of Appeal.

And then Judge Herbert Dixon, who's a senior judge at the DC Superior Court and Judge Xavier Rodriguez, a district judge from the Western District of Texas. And then Judge Sam Fuma, who's on the Arizona Court of Appeal. And then we joke it's five judges walk into a bar and we run into a computer lawyer, computer science professional. And that's the one keeping us all honest. And that's Dr. Maura Grossman, who's a computer science professor and also a lawyer. And also a guest of the NeLI podcast as well. And actually, I met her through NeLI. That's how I met Dr. Grossman. And she's become like a very important collaborator for us and helps us get judges trained on Gen AI and eDiscovery issues. So I'm very grateful for that connection that came initially through NeLI.

But this group, I got to know Judge Schlegel through, we have kind of like an informal like email chat group, the group of judges from around the country. And it's not too big. It's about maybe seven or eight of us. And as the AI news started to come out, we just kept sharing information with each other. And that's how I got to know Judge Schlegel. And then when we were all kind of, he's the one who said it first, but everyone was like, I've been thinking the same thing. It's that we really need some sort of a guidance for judges on how to use Gen AI. The ABA had come out with their statement for how lawyers should be using it or things that like guidelines for lawyers.

We thought we need to do this for judges. And we wanted to do two things. One is kind of set up some guardrails that we can't believe like the judicial systems haven't put in place quite yet. Some have some haven't, but we don't have any firm federal rules about how judges should be using AI or Gen AI in chambers. And the same goes for a lot of state courts too. So we thought, you know, as a peer group, we should step out in front of this because not only do we wanna say, these are some important guardrails if we're gonna use it, but the second is we want people to start using it. So we wanted the message to do across that you can use it responsibly in your work as a judge.

And it's not like using GenAI in a legal practice or in any way in a legal profession. It's not an all or nothing. It's not like a switch where if a judge uses it, they use it to make decisions for them and that's the only yet. There's so many in between ways you can use it to make your work more efficient and I think better. And so we wanted to bring that. That was the genesis of this and that's how we got to the guidelines that the Sedona Conference ultimately published.

Well, so one thing is that it's a practice point but it's also important for people to hear that judges are cognizant of that and it's the principle of human in the loop, judge in the loop, jittle instead of hittle, I don't know, we've got to come up with some kind of fun acronym for that. But as I use more and more tools in my own work, I could see how there's a risk of becoming complacent because they get better. A lot of the tools we do use are kind of rag based tools. So they're trained on actual bodies of legitimate legal authorities. And so we just, that constant reminder of human in the loop, which serves as one of the best practice guidelines for judges, but two, I would think lawyers are happy to hear we're focused on that because you don't want, it builds confidence in how we're using these tools within the system.

And then the second takeaway is just that they should be using it. Like I do a lot of training for judges and now I've done it kind of around the country and I just show them some of the ways that I can use GenAI in my own work. And it's really fun because you can kind of, you can hear the jaw drop over really simple things. You know, you would laugh to know that like if I ask Claude, and I'm just using it as a sample not advocating any particular tool, but if I show like how I might ask Claude to help me with like in writing, be like a writing partner, they're like that. That was so fast. I like I can't, and when you see it, when you see someone do it and you see it in a way that's like literally can help you do your job, you're much more likely to try that out yourself.

So when we first started, it was like, you can use ChatGBT to draft a packing list if you're going to Madrid next summer or something like that, which is, I mean, it's a little bit, sure, I mean, yes, that's kind of cool and nifty, but that can that really help me do my job. Well, how about I can upload a special verdict form for a trial that the parties have submitted or even that they're disagreeing with and I can get feed both of those into an AI tool and I can ask it to give me three options that make that verdict form more understandable for the jury. And then I've got three different options to look at that I can play with. And then if I like it kind of, but I need a little bit different, I could just ask the AI tool to run another draft.

Like kind of that's good, but I still think it's a little confusing or you took out the word negligence and that's actually a really important word in that verdict form. So you need to put that back in. And I think people are surprised to see the iterative process that you can engage in, but once they see it, I think they're much more likely to try it themselves or talk to their clerks about trying it. And in the federal system, that's a big issue for judges because our clerks rotate every year. The traditional model in federal courts is that you have two to three clerks depending on what type of judge you are and your clerks rotate in and out every year. Some do a two year term, some of us have some queer clerks, but for the most part they cycle.

So when I'm trying to advocate to the federal system, like we need to get these tools and we need to enhance our research contract with this provider to include their AI tools. I say, because guess what? Our next come September, the next wave of clerks is gonna be using a different set of tools or used to using a different set of tools than we use. So we need to make sure that you don't have judges trying to teach them how to do bullying research. So that's what I really, If it can do anything I would really like for judges to read these guidelines and say, okay, I can do this. First of all, there are safe and responsible ways to use it. I have to always have a human in the loop and it's worth playing around.

Well, we're in the middle of coming to the grips of it now, I think, because what you see is you've got these standing orders for a lot of courts and they're really hesitant about lawyers using GenAI, hallucinations showing up in the briefs that they submit. And I think there's just also this concern that if I type something in to Chatchapiti or Claude or a different type of program, and I'll default to Claude because it's one I use the most, but that's not an endorsement. Once you know something, it tends to be what you use. Plus it tells me good morning whenever I open it up. So I like that. It's very friendly with me. We all need affirmation. I do agree with you, Josh. I do agree.

Yeah. But so I'll just use that as an example. I know I hear from judges all the time and they say, well, what if I type in, I'll give you an example how I use it today. I had a settlement conference. I was frustrated because the parties weren't prepared. So I'm going to issue an order to show cause. Why? I shouldn't take certain actions unless they kind of clean up what they're doing. So I put in, Claude, I just typed in a bunch of things about what happened in the settlement conference. And I didn't have, it wasn't organized. It wasn't focused. And I typed out, I said draft an order show cause. And it did it really well. It was a pretty clean document. It wasn't perfect.

It was clean. And it was a good starting point for my clerk to, for she's going to do her run of it, then I'll go back and look at it again. But it definitely was faster that if I tried to like start from the beginning. And a lot of judges are really hesitant. I think what I would hear from some judges would be, but oh my gosh, what if someone else is in Claude later that day and says, what does Judge Goddard want Claude to do? Or that they could somehow go and get my prompts. But before I started to use Claude, I went and read the terms of service. And the terms of service say that Claude won't use my inputs or outputs to train the models.

So theoretically that shouldn't be happening. And I think that it's reasonable for me to rely on what the terms of service say. I'm not saying that companies have never breached those terms or done things they shouldn't have done with their customers' privacy. But I think that's reasonable. I also think you don't know that much about my judicial thinking. Except maybe you might read what I put into Claude. I gave you a quick summary. But when you, if you read what I actually put into Claude, I'd be like, well, she was really mad. Right? That would be, that would be all you really learned about that. That's new, right? That's not some special judicial secret sauce. And when I talk to the judges about that, I say, well, you sign warrants and you get people's Google search history.

Does that stop you from putting things in that search bar? Because you can learn a lot from a person by looking at what they're searching on Google. And that's not publicly available just like your prompts in Chatchapiti or Claude or et cetera, Jim and I would be. But it is, I mean, I guess theoretically it exists somewhere in the ether and could be, but I just don't know that it really tells you that much. So I do get a lot of, I do get some pushback from judges about concerns about that. And I don't think that's an illegitimate concern. It's right for them to be concerned. That's a really smart and good question to ask, like, okay, how's my information being used? And how could it come back to bite me?

But it's, I think there's answers to that where you still should be comfortable putting that prompt in. You just got it.

Yeah, I've used that same analogy. And so many times, but except for I take it a little bit further, it's not just a really bright and smart first year associate. It's one who always wants to make you happy and always wants to please you. And so even when they don't necessarily know the answer, they might make up one because they don't want you to be disappointed that they didn't know the answer. So that's a great analogy, but I always think of it as like, it's the most eager first year associate, right? I also think when you're like, we all get fatigued in our work and if you're working on something and we don't have a lot of stuff on settlement and which is a lot of what I do, I don't have a lot of time to think about things.

It's really rare that I have in my calendar to do some settlement conferences, three hours in a row to like work on a document, right? And I kind of religiously guard that time and my clerks will book out time for me in those big chunks because that is uninterrupted time, right? But that doesn't mean that I don't have a lot of stuff to do, even though I only have a short amount of time to do it. So if I'm writing like a quick order in a criminal matter, right? It's a really good and helpful thing to go and run a paragraph through an AI tool to make sure it makes sense or if I'm stuck from looking for an analogy, I'll just pop in there and I'll say, I have a lawyer who is just, everything is so dramatic and it's, but I don't wanna say making mountains out of molehills because it seems kind of trite, right?

I'll say, can you give me some other analogies I could use for that and I'll look to see if I can find that. And I think that helps, I mean, I like to think it makes for better writing. You could, I suppose, have just like an order that's just full of bromides because you're just getting them all from AI, but hopefully we're better writers than that and you try and find a way to kind of shuffle it in. But also if you're tired and you've gotta explain, like you could upload a brief in like where you have to summarize the party's position in an order. And there's some orders that I do all on my own, I don't use a clerk at all because I don't really need to research them because it's like maybe it's on the criminal side, it's a bond order and it's something I decide all the time and I know the law and the law hasn't changed.

But I do need to write an order that shows the parties that I've read their briefs, I've considered their arguments and this is the reason why I'm either agreeing with them or not agreeing with them. That process alone builds public confidence in the judiciary, right? Feeling heard and seeing tangible evidence that someone that you were heard. And so the way we do that sometimes can be challenging. If you're, sometimes you just run out of words, right? But I can upload the briefs into a tool and say, can you summarize plaintiff's argument? And then I've read their brief, I've read the summary, I tweak it to make sure it sounds right. And then I've got, but that's a much faster way for me to draft that order than to try and think of a different way to say what they said.

Because I don't wanna just quote, I just don't wanna block quote them. I wanna have an order of this information to read. So in that sense, it's more, it goes a little bit beyond just the first year, although the hallucinations I think I attribute to the very eager first year associate, it really is kind of like a writing or a thought partner. There is so much that I'd love to unpack with everything you just said, Judge Gatter. One of them, you brought back memories of when you're talking about creating this language for in order to show cause and that you just, you throw your thoughts in, you ask it to come up with it and it comes up with a pretty good first draft. And all I can think to myself is the days when we used to have to go to the library.

to pull out a book in order to find the template and then manually actually type into word what you're doing.

No, I think that's true and I think it's fair. But it also helps me demonstrate to them in a faster and more efficient way that I have heard them, that I read their brief, but I understand their argument. It's funny because I always joke, I started out at Cooley when I was first associate and I always joke that I had chat GPT in 2000 when I started and people said, I said, what was called Seth Raffkin. If Seth may, if you're ever listening to this, I hope you enjoy this. He's a lawyer who was a few years ahead of me and he was in my mind, he was the smartest associate at Cooley. And if I was gonna be successful, I needed it to be more like Seth.

And so we have the document, what did they call it, Docs Open? Anyone remembering this? You get in touch with you when we talk about it, with Docs Open. And so when I had to draft anything, emotion to compel, emotion for summary judgment, the first thing I did was I went into Docs Open and I looked for anything Seth had written that looked like that in a case that was similar. And I modeled everything I did after him. And so I don't see it as really that different from that. It's just turbocharged and I'm getting inputs or suggestions from more than just Seth. I like that. I'm gonna try to figure out a way to weave that into my conversations as well. So one other thing that you said I thought was interesting as well was this idea of just inputting the plaintiff's arguments, inputting the defense arguments, what have you, and then being able to have it summarize.

So again, you can focus it on the work. Now I clerked for a judge when I came out of law school as well. And I typically put all that stuff together for the judge, whether I did a good job or not, is something that she might have to talk about on another program maybe. But I'm curious, is it replacing the law clerks? In some sense, the way you had phrases you do some of these on your own, but is it a replacement for a law clerk or there's now a more efficient or a more meteor place for the law clerk now? Yeah, I definitely don't think it replaces the law clerk because I think when I draft an order, and not a lot of, not all judges do that, right?

Not all judges will start an order or draft an order from start to finish on their own without their clerks. And so it might be a little bit of an omelette. I don't do it all the time. I don't have time to do that all the time. I need my clerks to be doing a lot of this work. But the clerks can do it, use it that way too. And I have guidelines for how AI can be used in my chambers and my clerks are allowed to use Gen AI tools. They just have to let me know, and we had to talk about it to make sure I'm comfortable with the tools they're using. But I want them to use that too, because they have families.

I don't want them here 80 hours a week. I want them to do their job with integrity, but also with efficiency. And so I want them to have all the tools that they're, like if we, I wouldn't make them hand write their briefs once we had Microsoft Word, right? I mean, I want them to be able to use all the tools that are available to them to get their job done in an efficient way without compromising the integrity of it, which I don't think is required at all.

Yeah, so that's a great question and there's two different ways to look at it. There's one as a user, do I care, do I need to know? I mean, I understand, I've sat through plenty of CLEs about what an LLM is and I have some basics down about that these are in some ways, it's just like a text predictor the way, so and I probably, because I've sat through so many different CLEs on that or programs on that, I may know more than your average judge about it, but I don't feel like I need to know any of that to use it in the way I'm using it, like to use it for to be kind of like an idea generator to use it as like a thought partner for writing, things like that.

It would be different though, like what we're about to see is, and it's just coming to the front, I haven't had a case where I've had to deal with this now, but there's huge issues coming down the pipe for judges on how other people are using it and how that presents in the cases we decide. So for example, discovery, how is a GenAI tool being used in eDiscovery? Is it producing a fear result? Is it producing a complete result? Is it getting us to the reasonably responsive set of documents that this party should be producing in the case, right? We're not gonna ever be able to open up the hood and look under it or much less understand it once the hood's popped, but we're gonna do that evaluation the same way we did.

With TAR, right? The same way we did with your basic search terms. Parties are gonna have to submit expert declarations to educate the judge on what the judge needs to know. And I don't think it's gonna be a matter of having to, you don't have to become computer scientists. We never have. We won't, we can't. Sorry, more gross men is a unicorn. And like most judges just can't be so proficient in so many very different areas of the brain. We're issue spotters. And we are, our strength is in our ability to fly above the forest at 10,000 feet and look down and identify where the problem areas are. And so it's gonna be, we're gonna follow that model of you're gonna need to be able to show through expert testimony, whether or not you're gonna need to defend your work if you're the producing party or challenge what was done if you're the receiving party.

And case law will develop. Hopefully I would like to see that case law develop using GenAI and discovery. If the tools become more economical for smaller model cases, for smaller cases or smaller amounts of data, then you'll see lots of case law. But even with TAR, I don't think we have a ton of case law on it because I think it's just, it's almost, it's really expensive. And so it makes sense in a smaller number of cases, even though I think it would be very good technology if we could find a way to scale it down for the smaller cases. But so that's a long way of saying like, you're right, we don't know what's in the black box. We're gonna have to figure that.

The other thing, that's GenAI and discovery, but GenAI and how companies are making decisions, yes, you're gonna, the companies who are using a GenAI tool to make decisions that could draw the attention of a class action lawyer or federal agency as far as discrimination is concerned, they need to be ready to defend that. There needs to be some way to test the AI tool, even if you can't look inside and see, explain exactly how it's doing it or why it's making a decision that you've tested, you've bumped it up against several examples. The easiest one, I think for people who understand is resumes. If you have the same resume, but you change the name on the top to something that has the more ethnic sound to it, is it gonna yield the same or different results and the ways to test that system?

So it just depends on how the AI is being used functionally to how deep you have to go into the black box. But expert testimony is gonna be really critical in all this. So now you've gone somewhere which I think is to me, because I'm a bit of a geek, I think this is kind of exciting. So it's kind of exciting, but also... Yeah, I know me too. I know, I know, I know. Now it's from one nerd to another. That's right. It's good.

And we're never going to get to the substance. Yeah. Well, let me talk about two different ways that I think we're going to see it. And I'll talk about the easy one and the easy one to solve. And I think that every document of production has a producing party and the receiving party. And I think that as the sophisticated litigants may have issues with how the receiving party uses GenAI to review those documents, which I think is going to be kind of interesting. And they may want to put some restrictions on what types of GenAI tools the receiving party is using, particularly if the information in that data set has been designated as confidential in any way, shape, or form. So I think there's easy ways to get around that.

But I do think that judges are going to have to be educated by the parties on why or why not whatever particular tool the receiving party wants to use to review those documents is going to use to do that, and whether or not it presents a risk from a security perspective. One of the things I don't see in protective orders, which just always surprises me, I hardly ever see it, is any sort of restrictions on how the receiving party is keeping the information. How are they? Is it in Dropbox? Is that OK? Are you OK with them using Dropbox? Is it being stored on a local server? I don't know. The parties go to all these links to designated other information is confidential. And then it seems like a lot of people are probably going to be in nightmare.

People are probably going to start putting this in protective orders. We'll be fighting over it. So I may have just hurt myself. But anyway, it's an important question, I think, in how that information is governed. They're going to put that information. You're going to hand over that information that you've taken all the pains to design it confidential, but you don't have any say over how it's being held, or even know or understand. That's one thing. But let's talk about that's, to me, that's a little bit easier as far as how you use GenAI tools. Because the reality is a judge is going to say, you can use that tool as long as it's not going to, as long as you can show it's going to have basic security standards.

OK. Now on the other side, the producing party, is a much for near issue. It hasn't come up before me, but I'm pretty sure it will. So I have to be careful about not prejudging it. But I can tell you, if you put that debate, if you wash it through our normal system, our process, I have rule 26 guidelines. I'm really careful. They're not just for ESI-driven cases. They're for all cases. And then you have to talk about ESI because I don't want, what happens in a lot of these cases is every case involves eDiscovery now. Every case has ESI. But lawyers in like single plane of employment cases say, oh, well, I don't have anything like that. I don't need to worry about it.

But sure you do. And if you, my rule 26 guidelines require you to go through those steps of talking about ESI, even if you may not have thought about it as an ESI case. But once you get to that step, I require you to meet and confer. If it's search terms, if it's prompts, I want you to meet and confer. Now, I do recognize I can absolutely see a case where let's say Defendant in a class action says no, I'm not gonna share my prompts with you. What I will show you is that we've done this collection and I can give you some samples or some either data or some statistics to say to try and bolster the fact that whatever your prompts were that you put into the black box, they can be trusted because you're yielding results.

So if you don't wanna share your prompts at all, you're gonna have to be ready to at least provide some information that gives your other side and the court some level of confidence. In the perfect world for me, although I don't know as much about this as the lawyers do, I've tested out two different AI, Gen AI Discovery tools through the demo process where you spend a few hours with it and you go through a sample data set. It was very interesting, very helpful. I may be one of like three judges in the whole country who is inert enough to do that if you wanna talk about geeking out, right? But I'm very hands on and I just wanted to see it. So they're not gonna have that experience.

You're gonna have to explain to them what that looks like. So through that experience, I know, like search terms and prompts are very different things. But if you don't wanna share your prompts, you better be able to explain that to the court and you probably should get an expert declaration on that because lawyers will tell us anything. Is there an obligation though to even demand prompts? I mean, there is an obligation. I mean, we argue over keyword search terms all the time.

Yeah, so you have to talk about all of that in my under my guidelines. And if you don't wanna talk about it, let's say if you say, well, I don't think that my position is I shouldn't have to tell you how I'm doing this. I should just produce documents to you and you should be relying on my signature under rule 26G to the responses that I've done a reasonable search and that's it. And there may be space for that. But one of the ways I would probably respond to that is, okay, well, if it turns out that you were wrong and your client has spent all this money on this huge process to produce these documents and then they are able to make a case down the road, either through a deposition of one of the executives at the company or through the marketing person and they say, oh, well, no, I had, we had these meetings and we talked to this all the time at these meetings and the plaintiff comes back and says, oh, we didn't get any of those, right?

You might have to redo things. So from a client perspective, from a cost perspective, I think it behooves you to have some discussion about that at the outset before you commit all the resources that are involved in using a discovery vendor and doing it your own way, and enroll in the dice on that. So I think you need to be thinking about, like, is that really the smartest way forward or do I wanna try and get buy-in from them so that then they can't come back and say later, oh, you didn't do this the right way. Then you can say, no, no, we talked about this, we went over it, I told you we were about to spend thousands of dollars to run this discovery process.

We relied on your agreement and that's where we are. You're gonna be in much better position with the court to say, you know, they bought it, you know, and they don't get to return it. And so I do think it's important. I think the measure, the amount of meet and confer should be, like, really dependent on how much you're gonna spend because you wanna protect that investment on, because it's expensive, this stuff is really expensive. So there's no one answer. I get nervous about, is there gonna be some court order that says, yes, they have to disclose all their prompts in response to a motion to compel or something. And then the question becomes it becomes taken out of context like but there will never be an order that says You do or you don't have to disclose your prompts and that should be the rule in every case It's just not everything is nuanced maybe you have to disclose your prompts because you've been engaged in such You know nonsense all throughout the discovery process and hiding the ball and it's almost like a little bit of a form of Sanctions that we have to employ to level the playing field and make sure everyone's got what they need Well, that doesn't mean in the run of cases people have to disclose your prompts, right?

You can just you can distinguish that case really easily So I don't think there's any one answer, but I do know it's gonna be a Big issue and who you have in that meeting them for is gonna make a big difference Do they know what they're talking about? Can they can they know where to give and where to stand firm? In it's it's a to me it's The sliding scale is the more you're gonna spend on discovery the more you need to make sure you make that investment wise from the outside.

Yeah Well, so we're as judges mean one of the best parts of our jobs is we really are kind of like one learners And one of the things that I just said personally in my own world The whole reason why I even knew what chat GPT was I teach a high school class on trial and the art of advocacy And I coach a mock trial team a high school mock trial team and I by three years ago, I signed an essay and I got the essay back and I will use it anonymity here to protect those involved, but I My child's in the class with the student and I said, you know Tom really turn to a Really good essay it was a little bit of a surprise, but that was it was good my child said oh That's chat GPT and I said chat what?

and My husband is a high school teacher and so he sat down and showed me and I was just gobsmacked like I have to understand this as a teacher but so that but so Having that sense of being like a lifelong learner and opening yourself up to like understanding what's happening in the worlds that you Don't live in all the time Is really important from the federal judiciary perspective as far as what judges are doing or how we're educating Each other really because it's a lot a lot of our in the federal system We have the federal judicial center runs a lot of educational programs, but they're run through it's a lot of peer Peer teaching it's judges teaching judges. So we go to like our judges go to judge school called baby judge school just to be cute, but I teach it at judge school and I teach eDiscovery and now that includes a discussion of Artificial intelligence, but we also have court webs where the FJC I've done a court web where I do demos for the judges We've spoken at the night circuit judicial conference and then we are magistrate judges have workshops We're really on the front line of discovering federal courts To workshops a year That's basically continuing education.

So even though we don't have to meet the bar requirements You all do which I've heard from some lawyers that maybe they feel will they present that maybe slightly But we still do Continuing education as judges and so on the federal side I've spoken at the I've done seminars on gen AI at the at the match or judge workshops last year I'll be doing the same thing again this year So we are really working to at least have some familiarity in the judiciary But that doesn't all what you can never have that deep dive that you often need Through the use of like expert testimony, especially with emerging technologies both on the discovery side and on the evidence side But I do I know I listened to once I had that Once I once I learned about what to have GPT was And I had my initial like I remember I talked to a friend who's an English teacher in high school And I said but how are people ever gonna learn how to write?

And you hear that from a lot of judges. Like, no one's gonna know how to write anymore. And he looked at me and he said, we are like math teachers in the 70s when they came out with the scientific calculator. And it's more that no one would know math anymore because this calculator did all the math you needed, right? We're just gonna learn in a different way, right? And I think that you have to kind of approach it that way and our clerks are gonna keep us educated as well.

So I, in the further judiciary where I work, I don't see it changing the chamber's relationship really a lot. I think we'll still have our clerks. I don't think it's gonna replace any of the clerks. I do think it's gonna streamline our work and make it more efficient. I can already see that in my own experience. I think that I would love, if I could have a wish list kind of, I would, it would, we, you'd start to see scalability for some of these tools, particularly in eDiscovery and in reviewing documents so that small and medium sized law firms had both an incentive and an ability to use these tools to make their own work more efficient. Cause I think it's not just efficiency. Like I think it's improvement.

I think that it can be a huge improvement. A lot of people are uncomfortable with that necessarily. Like, but when you think about, when you go back, I don't know about you two are younger than me, but when I go back to my coolly days, when I wasn't looking to see what Seth Raffkin did, so I could basically plagiarize his work, I had a stack of bankers boxes in my office and they were, I would pull one out at a time and I would spend an hour going through the documents, flagging the ones for review.

So for a while, right? And how many times did you get through your second box when you realized there was this whole set of documents that you kind of passed over, but actually they probably were relevant, right? I mean, how's that better than the machine learning techniques that we have? It just can't be, especially when you combine both, right? What if you have the human backing up and the computer doing a lot of the grunt work? So I hope to see more of that and I hope it trickles down to the firms that are smaller, sole practitioner, medium-sized law firms, because I think their clients could benefit from them using that technology, but it's gotta be cheaper and it's gotta be more accessible and they have to have an incentive to do it.

And then the other thing I would, this is, I don't know exactly what this looks like or how it happens, but I'd love to see some way for Genai tools to either back up or help take out some of the, it's a Trump judge, it's a Biden judge, it's a Clinton judge, it's an Obama judge mentality and judicial opinions, where there's some way you could bump up like what a judge says against an AI tool, maybe it comes up with the same thing, maybe it's slightly different, I don't know. But just to show like a, like almost a way to kind of test like the objectivity of some of the decisions you see because we're overly simplistic in how we cast what a judge decides.

I think that's a reflection of our society and a very highly divided society and we wanna put things in buckets because that helps us make sense of them. But I'm hoping maybe it helps us transcend that a little bit. We'll see, I don't know exactly what that looks like. But the other thing that's gonna happen is parties are gonna get really data hungry. And as soon as it becomes easier to review all that data, they're gonna want more. So that's, I don't, yes, I think there'll be some efficiency but I think it's gonna cause some information governance how to express, you know. If you can start applying, if you're the receiving party and you can't get the data, you can't get the data, you can't get the data, you can't get the data, you can't get the data, you can't get the data, can use a genetic tool to more easily digest what they produced.

I used to, when I did Plainness work, I didn't want a bunch of documents. My discovery requests were very laser focused because I didn't have time to review 100,000 pages. But now, if I've got something to help me, I want as many pages from that defendant as I possibly can. Oh, the Plainness. That's going to cause, everyone's still going to have a job, I think so. Well, that's a good part of it. No one has to lose their job over. We just learned how to do our jobs differently. Yeah.

Well, my pleasure. Thank you for having me.