A Conversation with Retired Judge Paul Grimm Part 2

It's great to be back. Thank you very much, Daniel and Jared.

Yes. I think it's important when we talk about AI. I still think that although we has heard so much about AI for the last five to ten years and generative AI in the last couple of years, most people, myself included until fairly recently, we really couldn't give you a really good explanation as to what AI meant and what the types were and how it was trained. We knew it had something to do with algorithms and code and that it was a technology that was adopted by so many platforms that are using it right now. And that's sort of where it was and they kind of left it in that regard.

In a humorous moment, I thought that Time magazine should be, make its person of the year generative AI because starting in January of twenty twenty three, I mean, just constantly, all you saw was generative AI and the specific type of generative AI, which was ChatGPT.

And there's a lot of reasons for that. I was talking interviewing, Maura Grossman this morning for my podcast and we were talking about deepfakes and she was really sort of talking about generative AI first became a matter of discussion, among tech people in the mid two thousand fifteen, two thousand sixteen.

There were some technical advancements that were made in search that made it easier to use generative AI in a way that did not require so much human interaction to be able to train it with the thumbs up, thumbs down kind of a thing. And then, of course, with chat GPT three point zero initially, made it available. And what it did that was different was, you know, where most of us could say, okay, well, if I use this software, it's got some sort of AI thing in there. I don't know how it operates.

But with the generative AI, I can go on there and for free, I can put a prompt and inquiry in there and ask, you know, what are the five strongest reasons why I should do a, b and c? And it would spit something out in real time. I mean, I've, we've all done that. You look at it and you see that your answers are starting going beep beep beep beep with the bullet points.

And it's so absolutely amazing and seems so plausible and persuasive. But what you don't really realize is that generative AI and these other large language models, they're not designed to find the right answer.

They're designed to predict what the next word or sequence or image should be based upon access to the data that it looks at in order to predict what the next response is to the queue, to the query that it got.

Now, the source of that is the entire internet. And of course we know that the entire internet is filled with things that are accurate and helpful and things that are, you know, conspiracy theories and things that are not helpful and accurate.

There's still to most of us is some sort of a black box and because we can get our hands on it and put information in there and it gives us something that looks so persuasive, it's beguiling. And that's where the competency issue comes in.

And, you know, forty out of fifty States have said that you have to have the competency in order to do it. Now that cuts both ways because there are definitely good and promising potential for generative value. I'm not, I'm not sure we're there yet. We may be close in some instances, but we're still not, at the point where we can say, for example, that we have currently developed and tested generative AI applications that can give us some of the benefits that we're looking for.

So let me give you an example. At Duke Law School, one of my colleagues, professor Jeff Ward is the director, executive director of the Law and Technology Institute at Duke Law School and he's a professor. He'd be a really great person for you to talk to by the way. Very, very great guy.

He has started this RAILS, R A I L S initiative, which stands for Reliable Artificial Intelligence in legal services.

And the specific goal of that Daniel and Jared is to do exactly what you say, to try to come up with ways in which we can address one of the most enormous problems that our system faces right now, which is access to justice. The the average American, and this goes through people who have good jobs that pay, you know, decent salary and benefits just just can't afford legal, assistance and are increasingly, disenfranchised from access to justice on things where it really matters to them. You know, you're gonna be evicted. You know, you have a, you have a landlord tenant problem.

You've got a nuisance that you need need to know what to do. You've got a, a domestic issue that you have something you need to have some help with. You've been denied credit or your credit report shows that, you know, there's some charge on there that you didn't make and that you gave notice of that. And you're wondering why your credit is bad or you've been turned down for a credit card or for a loan.

And, you know, these are things that touch just a huge number of people. And when they're polled and they find that they have a problem with that, that kind of a problem, they can't get it resolved. It just, it hangs over their head. It's just like an awful dark cloud that hangs over their head.

So, you know, the ability to have a program where you could access it and have this generative AI give you accurate information because it's been trained on data, which is accurate. So imagine, you know, all of the Internet is your source. You're gonna get a different outcome than if you have a company that trains its generative AI search tool for all reported cases in the state and federal courts. So they're all legitimate cases that really exist.

And of course, it's signing them for the right proposition. It's not a hallucination, which is using a case name for a proposition that the case doesn't stand for. Then you may be able to test it and get accurate results. That's what they're trying to do.

It's it's a big tent enterprise that anybody who's interested in this area can can be part of it and help, identify the initiatives that they move on. And I think that that has great promise, the ability to summarize documents. Imagine if someone had to respond to a motion for summary judgment and needed to have some sort of a synthesis of what the key document showed or even a judge. In fact, I know a judge, a federal judge is a very, very good judge.

He explained this in a bar conference in front of a, of a live audience.

The, the judge said, look, I have a bench trial and I had eighty witnesses and I have a huge record and I'm gonna do a little experiment. My interns and law clerks and I are gonna prepare our findings, facts and conclusions of law based on our review of the record, But then we're gonna test our work, but we have gotten access to an AI, legally trained AI platform that's supposed to go in there and read all of this record. And we're gonna see which one because it's gonna, it's gonna run ours and then get our results and then compare it to what it does. And then where it differs with what we do, we'll go back into the record and see who got it right.

That's kind of an interesting proposition and imagine scaling that up to where you could do it often enough and make revisions and corrections in the training to where it got to be helpful. You can see how a tool like that would be very, very good. Or if you had a complaint for employment discrimination and you were age you know, you're alleging age discrimination, do you wanna know what the elements of that? What did you have to allege?

And and you put your your facts in. You probably have promise in the use of generative AI tools to analyze a brief opposing party filed opposing your memorandum to see if they left out key cases or a judge could do that with the briefs that were filed by the parties. Did they address all the critical to controlling case law here? Or do they miss something?

Is there something I need to see? You know, all judges have had that experience myself included where the parties are talking and they're arguing and they have not discussed something that's really important for resolving that. Maybe it was a new case or maybe it was something out that they just missed it. And so there are potentials that are there as well.

Generative AI has the ability to be used in ways that I think some jurisdictions and some countries are using it. I think China's been doing this a lot where, certain kinds of claims. We would think of a small court claims. You know, each side can can go in there and put in their version of what what the dispute's about.

And the AI adjudication technique would come out and analyze that and give you a recommended resolution that it believes. And if the parties don't like it, they can go to a human judge. But the people who I have heard explain that say that eighty, ninety percent of the people like that, and they get it fast, They get it immediately. Talk about access to justice.

I know that Justice McCormack, the recently retired chief justice of the Michigan Supreme Court is now the chief executive officer of the American Arbitration Association. And she's really forward leaning and been very, very front leaning on taking a look at how you could use artificial intelligence platforms in the dispute resolution area.

And JAMS, which is another well known mediation arbitration organization is looking at that kind of technology as well. I think Daniel and Jared that the promise, the plus, right? The utopian aspect of this is not that far around the corner and there's money that's being put into this. And so that is great, but it's not any good unless you can test it, validate it and show that it's producing results that are at least as good as what you expect from, you know, lawyers doing that. You know, it's not fair to insist from technology levels of perfection that humans can't get to themselves. So you have to have some appetite for some error. So that's the good part.

Yeah. I think that one of the things is the challenge.

I mean, people didn't understand what technology system review did other than just sort of the generic discussion about it. And and you'll recall, you know, even though the technology people said this can really give you recall and precision rates of accuracy much better than a human being can do and at much faster and can be scaled up no matter what the sampling is until, you know, Andy Peck wrote his to Silva Moore case and said, hey, this is okay to use. People were afraid to use it. They were afraid to use it for the very reason why if they use it, the other side would say you use this whiz bang, shiny, bright new object that we don't know how to do it.

That's no good. Go out and back and do it the old way. And now they've doubled their costs, maybe even tripled their costs. Well, we got beyond that, you know, it took a long time.

But the trouble is, is that there's still a lot of debate about sort of how these individual tools that are using this technology do that and how we can determine whether they produce accurate results and valid results still is a mystery to most people. So what you tend to get is you tend to get a platform user and they say, well my system will do X Y and Z. And there's no sort of like common features that you can compare. So that, you know, like NIST, the National Institute of Standards and Technology came out and said, these are the hallmarks of a document review system that is sufficiently valid, meaning it produces accurate results and reliable across broad use and application.

Instead, each one has sort of its own secret sauce that it uses, and it will promise the world and a pony too. You know, maybe someone actually uses it and tests it and tries to compare it. How would you know? They make promises of accuracy and the entire goal is to, you know, we will use this as a resource that we retain these people to do and we are now offshoring that to them to do it.

Now there's some platforms that are very well respected and they have proven themselves and you don't hear a whole lot of complaint about their being used. But now if there's some new aspect of them, as you said Daniel, we've to make it even faster and better and to find these key documents, we've used generative AI and we've gone in there and we've looked at it and we think this is better. The problem is that when the other side says, well, how do I know that? I mean, you've got the records.

I don't have access to them. I've got to get you to produce them to me. How do I know that? And and now we get into the battles that we've still we're still fighting and they're not helpful battles.

Well, what's the seed set? You know, let me let me see how you tested it. What were the hits? What were the rejects?

What's your recall and precision rates? You know, how have you decided to do this? Just like battles over keywords. I mean, the truth of the matter is that all of these tools are potentially very, very good if used properly and iteratively.

But when you take something which is probably, you know, the average judge, and I include myself in that generalist judge, doesn't know how this stuff operates. We've been told that this stuff has promise. Who do we rely on to do it? And we bring in ESI experts and you still have this dynamic where, you know, it's not unusual in the case that one side brings a person who's, you know, sort of fairly well regarded in the in the tech search area and they come in and say, no, no, no, no.

This is not gonna be effective. It's gonna be too expensive. The other side brings someone else in and says, no, no, no. It's gonna be perfectly fine.

And we go up ended up back there again, either trying to bring into court, you know, the tie breaker as a technologist or the court trying to decide. And, you know, you've all seen those opinions where some of the judges say I'm the last person you want to design your keywords. On the other hand, some judges say, well, I'll give you the keywords.

And the point is, is that for each case, if there's no one size fits all, it requires that collaborative approach and sharing information and testing it and coming up with, say, okay, let's get a confidence factor. You know, that's like a black box experiment, you know, where you can say, I don't know exactly how it works, but I can I have some samples where I know what the right answer is? You can build the sample set that has the right answers in it and also has the outliers and run it through there and see what it produces and at least get some sort of confidence. If you had that kind of credentials that you could show, I think a lot of judges would say, you know, we don't have to have the best be the enemy of the good.

That's good enough for what we're doing here. But I think we're still fighting that because the technology is changing. You know, what's gonna happen is that, and this is something I think is problematic, is that when you license this stuff, the licensor is saying, don't turn over the source code to the other side. This is our, you know, our trade secrets.

And that's true. And each group that thinks that they've got the right finesse to be the best thing out there in the market are gonna be reluctant to just open up the, you know, the blinds and let everybody peek inside.

I agree with that. But that's not any different than human lawyers right now that I'm sure you see every day in your courtroom who come in there and they make stuff up. You know, I'm sure that they do. Right?

They can read all the good case law that there is. They still stretch the truth, I imagine, in your courtroom every single day. And so is that any different than the what generative AI is doing now? And how do you, as a judge, you know, sort through that and, you know, get to the truth and, you know, really sift through what is the source of truth there.

I think that there is sort of a double standard. We expect more from technology than we expect from humans. When I started, I got age on you two guys, but you know someone would give me a request for production of documents and I would respond.

No one ever came back and said, so Graham, tell me how you, how did you approach doing this? I mean, I want to know, you know, where did you look? You know, what boxes did you pull out? How did you know that you looked at everything in the box?

I mean, was there some sort of a numbering? Tell tell me how to do that. You know, I would have been doing cartwheel screening, this attorney work product, you know, you're entitled to have the information. If you can't show I didn't do it.

And what happened was is that the explosion of BCCs and email chains allowed someone like, you know, Judge Schindlin to have the case in Zuba Lake where the defendants were saying, don't have it, don't have it, doesn't exist. And a third party had it. Oh, isn't that interesting that you say that because look at this.

And that's when, you know, the sort of scales fell from our eyes.

But part of the problem is that what is sufficient, what is reasonable and that is still under the rules of procedure, a case by case matrix.

The problem we have is that I think many times we're talking past each other. You still have those impulses where the requesting party doesn't know what the producing party has. And so the way you deal with that is by being over inclusive.

And we still see what the twenty fifteen changes were supposed to prevent, which is boilerplate objections. It's overbroad, it's burdensome, it's not proportional. Now, Daniel, if you and Jared are, you know, in your law firm and you request documents and someone comes back and says it's overbroad, burdensome, and not proportional, and you are acting in good faith to try to tailor it if you can, you have no information about how you could tailor it. You have you have no idea.

And if it's a dispute, then the judge has no idea. Are we talking about, you know, one person's definition of proportional is another person's definition of inadequate.

We still are sort of fighting these foundational fights all over again.

And maybe it's because the technology changes and so we sort of find ourselves with new technology and going back to what we're familiar with and doing that.

And I'd be interested in your perspective because you're the ones who are doing this on a on a case by case basis all the time. It sounds to me like we're still not in that promised land where the kind of collaboration is starting off before we start to actually get and exchange the documents so we can move on without the kind of time delay and resource loss that we have all tried to avoid.

Well, Daniel, if I may interrupt you for a second. Every time I see one of those news stories pops up, I wanna know how many lawyers have been sanctioned for the exact same thing not using generative AI because that never hits the news.

But, you know, what I think is that when professor Grossman and I do one of our ABCs of AI, We talk about some of the biases that surround technology. Right? And so there's a heads and a tail. So you've got technology aversion bias.

So it's like, there's no matter what this thing does, I don't trust it. Am I gonna get on that autonomous airplane? That airplane has been tested and it only crashed, you know, once every thousand times. And in real life, they'll crash five times every thousand times.

So I'm a much safer on one of these, you know. But am I gonna get on that with no human being up there doing it? The answer is no. I'm not getting on there.

Those kinds of biases figure into it. And then part of the problem is that technology changes so quickly and is described in ways that that are so phantasmagorical that we can't really understand how it works. And now we use this stuff without even worrying whether we know that it works. And that's when you can do something that most of us would say you would never have thought.

You would at least look at it. At least you look at the head note for the case before you sign it, and yet here we are.

Yeah.

Well, and I'll tell you the other thing too is that, yes, but it's starting to happen. It was last week, I think, that someone sent me a intermediate Missouri appellate decision where a pro se litigate had used, some of that stuff and had been warned against it by the court and yet did it. And the case to respond to a summary judgment motion. And not only did they lose the case, but the judge sanctioned that individual ten thousand dollars. And among the authority cited was the article that Moore and I wrote that says, you know, you have these ethical obligations regardless of whether using technology or not.

I enjoy it. You guys gotta come down to Durham and I can we can continue this, over a good meal and have a good conversation.