Paul McVoy on eDiscovery From the Paper Days to AI

Thank you so much for having me, and for the opportunity.

Gosh, the most interesting thing I think I've seen is the advent of people using a computer's brain to leverage what it can do over humans. So you've mentioned paper. And and I mentioned paper so often when I talk about this stuff, because when I did my first second request. we had about 1,200 boxes of paper. We had to hire a team of 9 people 8 h just to load the paper into a semi, and the first year that was in charge of the second request, when I worked at Jones Day was a little paranoid about rain, even though it wasn't supposed to rain. so she sent us to Home depot at 4 o'clock in the morning to get a tarp to put over the semi.

In the event it might rain before the next morning at 8 Am. When we had to deliver it to the Doj. So you know, when paper ruled the day, my life was a lot harder than pushing a button and copying things to hard drives, even though hard drives fail and give me more white hair. But you know.

That's a great question. I really haven't thought of it in quite that way, but you know I was fortunate. I worked at Jones Day, which is a huge firm that works mostly for corporations and defense clients. And then I went in house at a vendor, and then I went to a primarily plaintiffs firm. And so I've gotten to see a couple of different perspectives. And I think in the vendor space we're much more open to trying new things. to leveraging new technology, to differentiate ourselves as leaders in the law firm side on the defense side. It's more of a traditional classic kind of approach. They're not trying to forge the new way to practice law or the new way to do discovery. They're doing the tried and true.

And then, on the plaintiff side, you have those plaintiffs lawyers that are trying to, because plaintiffs pay for everything that they do. They're trying to leverage computing power. They're trying to leverage anything, to make their cases more efficient, and to get through documents and digest them in a different way. So I think each has a very, very distinct flavor. The vendor side. I like more because of the entrepreneurial aspect of it right if I have an idea, or if I have a thought of something that could work for the industry. I can implement it and then try and convince people to use it right on the law firm side your clients even on the defense and the plaintiff side. you know they pretty much dictate how the case is going to go.

The vendor can really lead in a different kind of way. I mean, obviously, we listen to our clients and do what they say to do in the best way possible. But we can. We can try new things, and we can innovate in ways that you just can't do in the law firm.

On the law firm side, I find that they're very risk averse right? So if you can bring them a hundred percent surety on a new solution. They are both feet in. They want to save money just as much as anyone else. but they also don't want to risk their role with their client, and because. I would say at least 70% of the time we're introduced to a client through a law firm. They're kind of putting their name in front of our name in front of their client. And if we fail, or if the technology fails, or you know, something bad happens in court because of what their vendor has done. They take the full brunt of it on the flip side. They also take the full benefit of it.

You know, when I was when I was at a law firm we would be able to save clients, literally millions of dollars, just because we leveraged early AI back in the days of a 10 x do either of you remember a 10 x. So when we used a 10 x, and we could actually really go 10 times faster on document review. Everyone was astounded. The Doj. The Ftc. Sent over their lawyers to make sure they could really believe the numbers that we were giving them in the timeframes we were giving them. You know it. It was something that the law firms are willing to do, but they won't take risk in doing it. you know. It's it's just that they won't put their name on the line for a vendors.

Experimentation, even though we would never put out something that wasn't 100, you know, vetted from our end of it.

Yeah, that's that's that's a really good way of thinking about it. And and I'm I always describe my role as logistics. I don't really do anything besides making sure data gets from point A to point Z, the cases back in the day, you know, when we were all starting, you could lose your case based on a discovery issue, right? You would not find or produce information you had to find or produce. And maybe it was because your client didn't understand what they had to do, or maybe it was because you didn't ask the right questions. But cases were won or lost. You know we all have those cases that we saw dismissed after you know, certain discovery issues were found. But that's not the case anymore.

Judges understand that this is very tough, and as data grows and becomes more complicated, judges have a lot more forgiveness for people that are trying right. And so I always say to my clients. Communicate what you're doing. We don't have to get agreement with the other side, but we do have to communicate what we're doing, and we have to do it in a defensible way. So, for instance, I'm a big proponent of keywords of search term hit reports of null set testing right. I feel like it's a scientific process that starts at one side goes through and you prove that you're right by testing your results right? Scientific process. But some people don't believe that they believe that they should be able to develop their own keywords.

run them, and then produce the results of those hits. After review, and I respect both positions as long as you're transparent about it. right? So I don't want to just get documents over the transom and just have to try and understand why they're there, and why certain things are missing that makes no sense to me, and regardless of your position, and regardless of how conservative you are with your transparency. I think you're right. Communication will cure a bunch of the woes that we have experienced or the reasons that we're in front of judges. It just, you know, as long as all the parties are on the same page. I'm involved in a case right now where the other side does not want to do a whole bunch of exchange with keywords.

They don't want to have our input on keywords, and that's fine. You know. We've tried to get it. We let them know we tried to get it, and if the case goes without a hitch, there's no problem if there's a hitch, because their keywords were wrong. Well, then, we have grounds to say, Hey! We tried it failed, and now we can kind of impose our will as opposed to giving you a choice in the matter. But it's all about risk, and you're right. Risk makes a lot of sense to take in certain circumstances, especially if you're more transparent.

Yeah, I try to do that with our clients. I try to leverage my experience over time with how it goes. Better. Once they do that communication and that transparency, I mean, I'm all for a vigorous defense of your client of vigorous prosecution of your case. and I don't think that means opening your kimono on every single issue, on every single matter. But what I do think it means is that you understand what you're doing, why you're doing it, and why you're asking. I agree with you that parties that don't know what they're doing tend to fight harder. And there are some parties that know what they're doing and fight harder as well, because sometimes that works. And I just try and get all of the issues onto the table.

I mean you 2, as lawyers know this better than anyone, you have more time at the beginning of the case than you do right before trial. You have more time to look at things. You may have more time to think about things and strategize things, and I always say, you know, let's come to agreement on all of these issues. I'm a big fan of the Esi Protocol. I know it's a, you know, a thing that's falling out of favor. But let's have the fight. Now, when we all have time to go, let's just come to an agreement on what the issues are. Go to the judge, have them decide, and then we can all just move towards the case with a fully focused mind. I don't believe in the back and forth.

I don't believe in the motion practice. I don't believe in all of that on issues that just don't get to the heart of the issue. Right? Let's just agree. If we're gonna agree to disagree on search terms. Let's go get the judge to sign off on whatever approach it is, and we'll go if we want to use technology assisted review, and you oppose it, or you want different kind of validation than we're willing to give you. Then let's just go. This isn't an emotional battle that we have to have right. I am involved. I was involved with the flint water matter right in the Flint water matter. There were a lot of emotional issues. We were actually helping to make up for a big, huge mistake that happened that will affect people's lives for generations to come.

And that was a battle worth fighting, but it wasn't worth fighting on whether we got the governor's text or we didn't get the Governor's text. That is a point of law that a judge can decide. I don't need to have, you know. Tense meeting confers with the other side's technology expert to tell me that it's not possible. Well, it is possible. Here's the technology we suggest. Let's move forward or not move forward. But we don't have to fight. We just need to get to the answer. So I'm all about. you know, putting those people aside, letting them fight back and forth, letting them have the 6 month discussion on whether we're gonna agree that we're going to turn over custodians or not turn over custodians, or we're going to disclose whether we're deduplicating by custodian or globally fight about that until you're dying days.

That's fine. We'll have gone through 7 of our cases on our side.

Yeah, we try and do it by that open communication. I don't know how many times I have to fight to be on my clients meet and confer calls when they're about eDiscovery issues that they don't have a in-house eDiscovery expert or resource. I don't want to call myself an expert. But you know I've been around the block. I know what the issues are. I know how to issue Spot, and I know how to talk through them, or at least point out what my client needs to think about. Back in the day. I can remember they used to. because I'm not an attorney. They would sit me outside the meet and converse, and then the partner would send the associate to ask me what to say when they were doing discovery concerns.

And and it's kind of like that. Now, being on the vendor side. Your earlier question about you know, which side do I like better? I like being involved, and whoever's going to involve me the most is how I think I can be most beneficial. And so, you know, I try to stay involved. I try an issue spot. I try and bring things up to our side that they can then bring up to the other side. If we see issues. I mean a low hanging fruit. One is review is taking longer than we thought. So our production is going to be delayed by a week, you know that's not controversial, and if you haven't delayed, for you know I was on vacation, so I couldn't get back to you.

And then, you know, this happened, and so I couldn't get back to you. And now we're 7 months late, those are. That's 1 thing, and that I can't prevent that. But if there's a technical reason why we couldn't process data and get it out the door. That's a simple call to the other side that alleviates so much of the stress and the pain that happens.

Yeah. The Sedona Conference is extremely important to me. I feel like I'm I'm going down Memory Lane. But I was at Jones Day when Jonathan Redgrave, who is one of the founders of Sedona, worked there, and I remember him coming back from the first Sedona Conference, or what I believe is the first Sedona conference. It might have been. you know. It was a planning session for this thing, and he came into a room where I happened to be coincidentally, and he started talking all of this exciting, galvanizing. Let's move the law forward. Let's, you know, put some meat on the bones of what is going to become eDiscovery. We didn't use E. Discovery back then. That wasn't a term that we talked about. and I just remember sitting in my chair saying, I hope I can be involved with this, because.

as exciting as his ideas are, they totally are lacking on the technical execution side. So I went back to my team at Jones Day, and I drew a picture of a rocket. And I said, If I presented this to you and said, Okay, I need this tomorrow. There's no way we could do this. We could probably learn to do it. But just because we say it's going to happen doesn't mean it's going to happen. And so when I got involved in Sedona, I got involved in the technology resource panel. and we actually have a meeting next week in Phoenix as part of the regular Sedona annual meeting, and what it is a group of users. So a couple of people from defense firms, plaintiffs, firms, the government and you know, judges that sit on this thing that is also populated by vendors.

Right? So we have, I think, right now 9 different service providers that attend this meeting, and what it is trying to draw that or build that bridge between the 2 sides. The lawyers that actually have to go in and argue and get things produced, and discover how they're going to discover in the right way. In a way that's fair to their clients and fair to the case, and adheres to principle one of the Federal rules right? But it has to all be done technically, they can't do anything without the technical end of it. And so we try to bridge that gap, and there's a huge gap. So why, I'm so excited. So I'm I'm the chairperson of this technology resource panel. We're the ones that put out the glossary.

Right? So we get to actually define the words that define E. Discovery. It is the most downloaded document. even more so than the principles, the Sedona principles, which are the framework where they're trying to put the cooperation in place and to draw a framework that you can use to do your discovery. So we're more downloaded than that. We're more downloaded than the cooperation proclamation, which is a document signed by hundreds of judges saying, You know, parties, please just talk to each other. Just communicate. That's all we ask. You know we're not going to mark you down, and we're not going to think you're not advocates for your side. If you just cooperate with each other. So I'm very thrilled with what we've been able to do.

And and now that AI has kind of introduced itself in a broad fashion. I mean. you guys know, we've been using AI for many, many years in the in the eDiscovery realm. Technology assisted review even before technology assisted review when we were just doing manual kind of predictive coding, where we were running searches and applying coding based on whatever it was, what criteria it was. Now that AI is there, I think there's an even greater role for technologist to play in the formation of how the law is going. you know, and we can. We can help put the box around the playground that the lawyers need to play in as they leverage this very exciting technology.

Yeah, there's a lot of exciting stuff going on with AI, and if I had to boil it down I think it would be to the summarization level. I'm a healthy skeptic of AI, right? I follow a lot of people that do the development. I follow a lot of you know, people that are on the back end. Talking about AI, there was a case that I saw today about a judge who ran a mathematical formula through generative AI, because the expert on one side of the parties had done that, and he just wanted to validate what the expert had done, and got 3 different answers with 3 different questions on 3 different computers. And you know it goes back to you really have to know what AI is.

AI is. One of AI's largest flaws is, it can't do complicated math. So you would think a computer program should be able to do math right? It's like a calculator. It isn't because any kind of its source that comes from a wrong answer to a math question is just as true to it as the right answer. It has no discernment ability. and that goes across the spectrum anytime. There's any kind of I would say, flaw in its digested material which is hard to have unflawed human input. You know, you're gonna get that? The the iteration effect now, where AI is also sucking in its own output to learn on is like the old ditto machines. If you remember taking tests when you were in 4th grade, and you couldn't read the questions because you were the last one in class, and got the last ditto, and all the words were blurry, and you had to raise your hand.

What is this? A 4 or a 7. I don't know. I can't answer the question without those things, so we have to be very, very careful with AI and similar to when the World Wide Web came out and everyone rushed to find the great solution of what W. The Www. Is going to be. You know we've had 1,400 companies. AI companies go out of business in the last year because they have this great idea, and it just goes away. You know, I think we're still at the very, very beginning of what it's going to be right now. I'm very excited about digest, because I think it reliably can digest documents for you. It can reliably digest groups of documents, but I would not rely on it to tell me what my discovery 100,000 document Corpus was about.

Maybe I would do 10 documents at a time, 20 documents at a time similar to how we do grouping with non generative AI right group, the documents that you think are similar together, and I'll look at them so I don't have to look at all the time sheets, because you're right. These are all timesheets. But I do want to look at these documents that you are classifying as this. But once that classifier tries to go to too big a corpus. It just it gets too muddy. My I'm I'm actually making a movie. And I came up with this idea with Leah Bays at Delhi, as a matter of fact, and the movie is. you know, ideal generative AI is like a white blob of wet paint.

Right? That's your ideal. That's what you want it to be. That's the corpus you want to be working in. That's the paint you want to use to go over your black wall anytime any color touches it, even the smallest little color. You no longer working in a pure world. And so you now you have a little dot of red in there. You're now working in a pink world. So every answer is now tainted by it. It's not like in the predictive coding world, where you might have one slice of the pie be wrong because you had a bad coder, and he didn't do his issue right. The the disillusion of that entire set happens with one bad. you know. Input, and so we just had to be careful.

We just have to vet it in a way, and we have to come up with ways to validate it that we don't have yet.

Sorry.

Sorry.

No, I think you're a great predictor of what is to come. But I have such a different approach. Right? I have such a different approach. If you can show me that you validated whatever process you're using. you could have kindergartners decide what you want to produce, and not if you can validate it and prove to me that it's a good process. I really don't care how it was created. I really don't care what went into the to the black box in quotes. There's not anyone that I've ever seen in any legal case. Well, I'm gonna stop saying that because I don't want to be too melodramatic. On my own side. There are probably 1% of judges that will take the time to understand what happens in a generative AI model, how it's created, how it's, you know, fed, how it's maintained, how it's designed to take a prompt and execute the predictive response that it has.

They just won't pay attention. Then they won't understand. If we got so far into the weeds to prove ourselves right. and they shouldn't have to. It's not the law. They should worry about the law. But if I can come to a judge and said, yes, we use this generative AI to help us find documents, and this is how we made sure it was working correctly. We looked at what it didn't find. We looked at the topics. We reviewed a sample of documents that no one ever saw before. To make sure that we had focused the engine the right way, and we found no anomalies, nothing wrong, nothing responsive in the pool that we didn't bring forward. There's not a judge in the world that's going to say, okay, well, that's that proves it.

If there's nothing wrong with your process, if there's no missing information that you're obligated to turn over, then why would I question what you did in the beginning. It's it's like me getting in front of a judge and saying, Judge. I think the other side sped on the way to court, and they have corvettes, and I just think that on the way here papers flew out of their backseat because they had the top down and the other side, said, well, our top doesn't go down now. I have to prove to the judge what kind of car they have, and that this top doesn't go down, or the information I can get from the computer inside. The car said the top was up when they drove 900 miles an hour to court.

No. that is so outside of the realm of what a judge needs to worry about. In my opinion, sorry.

Yeah, I think it goes back to validation. And I also the CEO of Everla, did a presentation when they released or widely released their AI tool. And he said it. Exactly kind of how I'm thinking about it, he said. If you treat your Gen. AI. Tool, no matter who makes it as if it were a 1 LA. first year law student. and you check the things they do, and you can rely on it as being, you know, thoughtful in what they tried to do. But then you verify what they did then you probably won't go wrong, and I think that's the exact right answer. It is a learning algorithm that's doing its best, but can only do as good as what you've given it. And you don't know what it's been given.

So you have to double check and recheck what's happening with it. I think that judges will insist that either the lawyers or their representatives, and I say that because you don't have to be the expert sitting in front of the judge, you just have to have access to the expert. You know people like me are happy to go in and explain for you ethically. That's all you need. You don't need to know it yourself. You don't have to be worried that you're never going to understand. Gen. AI. You just have to have the resource that does that can explain it on your behalf. And so I think that judges just want a thoughtful approach. All of the judges I speak to, and thankfully, because of my experience and just the road I've gone.

I get to speak with a lot of judges, and I get to speak to a lot of judges informally. and they're very excited about anything that's going to make the discovery burden less on all the parties. Right? They don't want cases anymore stuck in discovery than we do. They want the cases on the merits in front of them that they can decide and get through their docket quickly. and they're very excited about the possibilities. But they're also very wary, because they see people just shortcutting what their obligation should be to vetting what they're doing. So I mentioned that the Sedona Technology resource panel meeting that's happening on Wednesday in Phoenix and a huge part of what we're going to be working on this time is our validation paper.

So from scratch. We're going to do a validation paper on how to validate different processes. It's going to be how to validate keyword searching how to validate technology, assisted review, and how to validate AI and not like it's not going to be a primer, but it's going to be. These are the things you should be thinking about, and these are the things you should implement into your workflow. We won't tell you what scores are necessary or things to that granular level, because that's up to you. and one case where it might be a certain score in another case that might be way too unobtainable because of the source data, but the validation, I think judges will very much rely on the validation to bolster the ethical responsibilities of any lawyer in front of them, arguing for the adequacy of all of their searches.

Right? If you can't tell a judge, you know we're pretty sure we got all of the paper that we needed to get, and we never went to the warehouse out in the middle of Alabama, because it was too far, and the bird was too great to send 4 associates there. The judge is just not gonna to believe that satisfied your ethical obligation for a due diligent search.

Yes, a hundred percent. You are a hundred percent. You are Technologies will come and they'll come faster and faster. I think. Jen, I is one of the aspects of geni that's most exciting is that it can help us learn what we don't know, and focus our attention on things that we can do better going forward. It might not have the answers. It might not be able to do a black and white, you know. Here is my motion to compel, because it, it, you know, has its issues with the fine tuning of what is absolute and what's not absolute, but it gets us to the ballpark of what we can do. and it helps us to develop the solution, which is, I think, most exciting, right?

I mean, as a as a independent person out of the legal world. You now can use Gen. AI to help you program an app and release it within a couple of weeks on the iphone. and that's all up to your brain coming up with the idea and then tweaking the system. It's a similar thing. So so I'm sorry. Long winded way of saying. yes, you're exactly right if we can validate whatever it is, because the tools will always change, and they'll always change not for the better, you know we do go backwards every once in a while a certain mistake will happen, and all of a sudden we can no longer use predictive coding. We have to use some other system for a while, while.

The judges reeled from the latest, you know. inexperienced person presenting a whole bunch of stuff that gets through some loophole, right? So, regardless of what it is. If you can validate what you have done, then you're all good. It's it's starts from the very days of keywords where I would have teams coming to me, saying, I can't imagine. I can come up with all the right keywords come up with all the right things to pull all of the important information. And it's it's that fine line that we're on now we've created so much data, there's no possible way to look at everything. There's no possible way to see everything, to find every document. So we have to do our best. and our best. in some cases using keywords will be come up with a list of words that you think are going to be good for the for pulling what you want.

but you can test it. It's so easy to test, because, you know, there is a knowable set of stuff you're not looking at, and there is a tried and true statistical way of seeing if that contains any of the information that you find important to take forward. regardless of how you got to your reviewable set, right the set of data that's made it up to the lawyers for physical review. There's always going to be a set of data that has it. and that data is always sampleable. We have so many people say to us, you know, we can't pay to index everything. I'm not asking you to pay for indexing everything right. If you have a lawsuit against the Federal Government, they're not going to look at all 752,000 employees email boxes.

But if you have 50 custodians. it's very easy to sample 5 of those custodians and look at their entire email box and the data that didn't move forward. And it's a lot less burdensome. And it puts everyone's mind at ease. It puts both parties at ease that they've come to the right solution, and it puts the judge at ease. You know you've done this in a way that makes sense, it's scientific, and it actually accounts for human error which now we'll have. Jen. I error. But you know well.

Yeah, you know, I've done a lot of conferences, and I don't know if I've done too many conferences. But I've done a lot of conferences, and NeLI was a breath of fresh air. It it felt a lot like a working group at Sedona, because it felt a lot less partisan. It felt a lot less directed to a point of view. It felt more like a discussion, I think, having it at the college was amazing. I think putting people in that environment re-fires things in their brain that haven't been fired some of our cases for 30 years. Because it puts you in that learning. It puts you in that open point of view or mind frame, so that you can really listen and hear and understand things or see different points of view.

One of the discussions you know, that I had there. Was about hyperlinked attachments, and that's a huge controversial issue in our world right now, what's possible? What's not possible? You know what should be done, what shouldn't be done? The obligations of the party, whether documents should be considered, you know, in context, or by the 4 corners, I mean, there's a lot to talk about. We could probably talk about that for the next 2 h, and still not scratch the surface. But on the other side of the argument that I have, which is typically a more plaintiff centered argument on this topic. You know there were defense lawyers saying, I totally agree with you. I agree that we should be working towards a solution, and that the solution is much closer.

Then, then, you know, it appears by some of the rhetoric that's going on, or some of the filings that happen, you know. And and so I. At the NeLI Conference, I had an opportunity to really communicate and exchange ideas with people of different perspectives that, you know, moved my perspective which you just don't see a lot in our world anymore. We're very divided, you know. world that has their own entrenched sides. And and being at NeLI was a. It was truly a breath of fresh air, because you could just meet with different people, talk to different people, get their unvarnished views in a safe environment. It just was perfect, I mean, so excited that repair could take part in it. It was our first year sponsoring, I mean, we're only a 2 year old company.

So it's not like we sat on the sidelines for a while, but you know we were very, very thrilled to be. Take part in it. I hope it's something that we'll keep doing going forward. I actually spoke at NeLI during Covid, and I was very sad not to be in the room, because it just is such a vibrant. active community, and happy to be involved.

So much.