Tom Gricks on Global Aerospace, the Growth of TAR and Generative AI

Likewise, and I'm sure you are. Appreciate the fact that there's only one so.

It's always a pleasure, Daniel, to see you, and Brandon always.

Oh, yeah, yeah, like everything with me. It was. It's it's just happenstance that it goes that way. I am not kidding you. I ended up at the Cmu in chemical engineering, simply because the guy that I was following through high school. who I thought was brilliant, went there. And I'm like, well, if he's going to do this, I'm going to try this right? So I go. I end up graduating with a degree in chemical engineering. and I go into work with draveaux engineers. It no longer exists. It was a company in Pittsburgh, an engineering construction firm and honest to goodness, Daniel, I had no desire at all to go to law school. I never even thought about being a lawyer. and then I looked around, and I'm like man.

The only way you get ahead at this place is to have an advanced degree. Right? And I'm I said, I am not going back to engineering school. They are way, too geeky for me, and I looked around. And I'm like, Boy, there are a lot of Mbas. I'm like, you know what? I'll go to law school. I like Perry Mason. Let's give it a go. So that is the amount of thought that went into me going to law school. and that's why that's why I started it. And I and I will tell you this. What I thought is was very interesting is really the difference between being an engineer and being a lawyer. And I've I've explained it this way to people before. When you're an engineer, you know, this 2 plus 2 is always 4, right when you're a lawyer.

2 plus 2 doesn't necessarily mean 4, and, in fact, oftentimes it's not even 2 plus 2. It was so different that it took me a while to ever bring all of that together.

You know, Brandon, I think I think you're exactly right. I think what I really did was came to grips with the fact that I could actually use my engineering techniques to get to legal solutions that I wanted to. Right? I could start to analyze. I could start to analyze problems in an engineering way, saying, This is, you know, I need to know all of the facts. I need to define the problem. and then I need to find a solution to that problem, knowing what I have as tools to do it. So I ended up finally saying, I can actually mold these things together by using my engineering kind of technique to get to a legal solution. And so that's that's really how it came together.

And thankfully, I was. I was this close to quitting after the first semester, this close? After that it was. It was after that point that I realized I could actually make the two of them work together.

Yeah, I will. And I need to preface that by saying, I want you to understand that I argued the first case in the country.

What you need to explain to Judge Peck. See, I was the first person to argue against the opposition. When I suggested I wanted to do TAR and I, as I've told Judge Peck he didn't have to argue with anyone. He was a judge. All he had to do was pull out that paper that he had written 10 years ago, and he kept trying to write. and he would then dust it off, and that became the Silver morgue. But Judge Peck doesn't agree with me. He will never agree with me. Feel free. He will have the same type of reaction when you talk to him. But so you gotta love like. When I told you this, all happened. This all is through happenstance. You got to love how this works right?

I'm sitting at Schnader. and we were represented.

Yep, it was the law firm, and we were representing Upmc. The University of Pittsburgh Medical Center. and there was going to be a big review. and they were technically represented primarily by New York Council. So at that point in time, I had started to look at technologies that would actually save time and money in eDiscovery against just a standard review. Right? So I'm reading all of this stuff. I'm kind of learning it. and it gets suggested by New York Council. In fact, not suggested. They actually basically hired a hundred 50 people to come in and do the review before they even talked to the clients. I mean it was. It was odd. And I went to the lead litigator, our elder statesman. and I, said, Paul, we don't have to do this.

I said. Well, there's absolutely no reason we should throw 150 people at this. We can do a much better job with some of the new technologies, and in that point in time there was more than anything else. It was TAR. One right, and Maura may have was probably out there. Maura and Gordon were probably out there doing their cal work as well, but most.

Don't agree.

Agreed. So so anyway, that's how I started getting involved. And through that case I ended up going to conference after conference and learning more and more and more about technology assisted review. And and I got introduced to a lot of folks who were doing technology assisted review at the time. and at the same time I'm practicing right. And I was working on a case with my good friend, John Stern, who's an aviation lawyer, and it dealt with the collapse of the Dulles jet center on 400 million dollars worth of aircraft. and he calls me up one day, and he said he said, you really believe in this technology assisted review stuff? And I said, I do. And he said, You want to do it like, Yeah, you bet I want to do it.

I'm like cool. So I gotta tell you it's it. Nothing ever goes exactly right for me. So I you know, I told John, I said, Okay, I'm gonna call these guys up. I looked at the data, Daniel. I'm sorry I can talk about this forever. But I looked at the data. And most of this stuff was not responsive. Right? We had like 2 million documents at the time, and at the time in 2011, that's a lot of documents, and I looked at it, and I said, There is. There's no reason for us to look at most of these. Let's use TAR. And let's eliminate this and get the documents out there and get this case tried because one of the other things that I did was I practiced in-house for 5 years.

and being in house, I have a real appreciation for budgets and reserves, and nothing can blow up budgets and reserves in a litigation like Ediscovery. So I said, Let's use TAR. So I called up the lawyer for the other side, and I said, Hey, look! I'm telling you there's not much. Here. let's do, TAR. I will not only let you have your nose under the tent, you can sit at the table with me. You just can't look at privileged documents. All right, I said. We'll make decisions. We'll make them together. We'll figure out what's good. We'll let the tool do its job. We'll make sure it worked. We'll validate, and then we'll go forward. And the answer that I got was. you know, when I do reviews I sit down and look at every document.

I just don't feel comfortable with this new technology. Seriously, I mean, seriously, you don't believe that somebody's going to say that right? And I said, well, if you're not gonna if you're not going to let me, I'm going to ask the judge. And so I filed a motion for protective order to permit me to do TAR. That's how that all happened, because I could not in the simplest of all worlds, which is, let's get these documents produced and try the case. His answer was, no. So that's how. That's how we got the global.

You know, Brandon, the answer to that is. if you want to know how I feel it. It did it. I feel it didn't do it the way it was supposed to. Okay, I can tell you this, I looked at that case, and what we ended up doing is at the end of the day, out of a 2 million document set we only had to review 200,000 documents. That was it right? So I was like man. This is absolutely fantastic. This stuff is phenomenal. This is, people are going to jump to be able to do this type of review to the point where I was so worried that people were not going to do a good job of producing documents and weren't going to adequately discharge their rule 11 responsibilities that I wrote.

I wrote the paper with Carl Schinnemann on. You know the implications of Rule 11 on that, because the one thing you know, when you do, TAR is you're leaving responsive documents behind. Right? So you have to be comfortable that you have actually fulfilled all of your obligations for production. And you do that by validation. And so I kind of went around and round about how you can do validation. And I think the nice thing about global is, it gave people a roadmap for things they could do if they wanted to. That was really, I think, what it contributed. It's there. The briefing was all available. You could go in and get it, and you can see how and why to make these arguments about the efficiencies of TAR and the value of TAR.

The results take away. Sorry, Daniel.

I do. I do think it's the general tendency of lawyers to be skeptical about just about anything, and as you are absolutely right as an engineer. It was so easy for me to understand. Because yes, it's it's a black box, right? It is a black box, because stuff happens that you can't see. But you had to understand that everything you do is very, very easy. You just look at documents, that's all you got to do. It's really simple, and at the end of the day you just test it to make sure you did a good enough job. Now that was my concern, to make sure that test was done appropriately to prove that you've done a good enough job so that you're discharging your obligation of reasonable inquiry in a production.

But you know I could I? Because I was an engineer. I had a hard time understanding why people didn't go with this. because it was so simple. And it's always been that simple. You make decisions on documents. They get coded. You test them, Ed. It was that easy lawyers, just I. I think there is a tendency to be risk, averse. and to not trust what you don't know. And and then we got into lots of fights about validation. And does it really work? And do I really see what I'm seeing, and all that kind of stuff? But but I do think that my engineering background made it a lot easier to accept this. I think we were talking about this when actually, when we were preparing for this, I mean.

when I was in engineering school. The one thing that I found out is that you can model anything. In fact. my senior project as a Chem. E in computing was to model the temperature distribution of a cancerous tumor. Given changes in blood flow and changes in external temperature. Right? So I'm like shucks, you can model anything you really can. And this is all about really smart people, not me, but really smart data scientists telling us how to build these models. And it worked. I mean it. It truly did work.

I think it was. It was fabulous. I mean it was. It was really for me. It was fantastic. It was. It was not easy, though, because when you know, with Tarwon. the difference is this, the difference is very simple. The difference is, you got to review more documents with Tar one than you do with Tar 2. Okay, that is almost a given right? So okay, I'm I'm really looking at Tar 2. And I'm saying, that's not a bad idea. I like what's happening here. But the way Maura and Gordon, the way CAL started it was almost purely relevance feedback. which means nothing more than I'm going to keep giving you things that you said you like. and my concern was always, if I keep pointing the ship just in the direction of the things that I like.

I'm going to miss all these responsive documents over here that I don't know anything about. So I was very concerned with that. Maura and Gordon got past it way back in the day by doing some measure of random sampling and making sure at the end of the process that those documents were being considered. Frankly, I got past it when I got introduced to the folks at Catalyst, to John Trudenek and Dr. Jeremy Pickens, who had created a CAL tool that incorporated an algorithm that would go look for the things you didn't know. I was talking to them about it. And I said, You guys have to understand. You've solved my problem. I said, that was my problem with CAL. And I said, You have solved my problem.

Now that I know that I am actually exploring the entire universe and looking for responsive documents that I may not even know exist today.

Sure.

Yeah, I mean that I was always worried that if you look, the point of creating a model is very simple. The point of creating a model is so that you can have that model look for things that look like that model right? But when you're doing a production, when you're doing a review for production. What you don't know is exactly what you said. It is what you don't know. You don't know what else is out there that has responsiveness has relevance to your case. But it's not something that you've been really looking for. So a pure relevance feedback model. We'll miss it.

And unless you happen to find it, because it'll poke around a little bit. But unless you happen to find it, what's going to happen is, you're going to run out of responsive documents in your review, and when you run out of responsive documents in your review batches, you know, you're pretty much done. That's true. With contextual diversity. Without contextual diversity you could never really be quite sure. So contextual diversity. I was a game changer for me. I thought it was critical.

It's interesting that you ask that because where the rubber meets the road is in my discussions with opposing parties that I'm producing documents to. because their question invariably to me is, Well, how do you know that you get this stuff? How do you know there may be stuff out there. And I said, You don't understand the way this algorithm is written. It is constantly going out for thematic groups and looking at them and trying to find the best document. Putting it automatically in my batch, we used 5%. We could. We could dial that up or down. But now, as soon as I've seen that document. I know that it's either good or bad. So now my model is considering the documents that I didn't know a minute ago, and it's pushing me ever so slightly in a different direction towards something else that I probably need or not.

So I've had conversations with opposing party where parties where they said, Oh, well, you've got to really do an awful lot of validation here and there, and look at all different places, because we never know whether you've seen all this stuff. And I said, the nice part about this one is this workflow. I don't have to do any of that, because it's automatically built in. And I'm able to. When I get through this. I know that the amount of documents that remain on any thematic group is very low. very low. So you know, we still validate. We still do all of that, of course, but at that point I don't have to worry about it. That's what I've talked to folks. I don't have to worry that I'm missing something because it's in there.

I love to hear all the people that say, Oh, don't worry, tars, tars. Accept that. Everybody does it. Yeah, no, they don't. It is still sometimes a fight. I know huge corporations that I've worked with that won't use TAR. and one of the reasons they don't want to use TAR is, they don't want to have a fight with opposing counsel about what's going to be in the Esi protocol. Well, and I'm I'm really. you know me. I'm I'm not opinionated on anything. but the fact is, the reason they don't want to have that discussion is because they just don't understand it. Let me have that discussion, because I can tell people how well this is going to work. and it will solve problems. It will get you, you know, we waste so much money on discovery, you know, it's interesting, because, like, I'm an eDiscovery lawyer, I hate spending money on eDiscovery.

I really think that we need to drive that down, and we'll get to that as well in terms of other ways to drive that down. But it's always been my goal to bring that cost down. and this is a way to do it. and if I you know I was. I was on a podium with on a panel, with a woman from ropes and gray at 1 point, and somebody asked her. What algorithm do you use? And she said, Listen, here's how I think I may have told you this, Daniel, she said. Here's how it works. I take all of my documents, and I spread them out into a big field, and I have a magic cow. and my magic cow steps on every document that I do not need to produce, so I pick up all of the documents that are clean at the end of the day, and I give them to you, and I can prove to you those are all the responsive documents.

Why do you care? Why do you care about my magic cow? And and that's the thing I you know. I know statistics are statistics, and you know I know people worry about that. But in reality, if you go back and look at what we used to do, I don't know about you, you know. I sat there and flipped through folders of documents. And you know, sometimes my review went. There's nothing in there just like it. It didn't look like it was a folder that was gonna matter. So I didn't look right. You got a worse job without the technology than you will get with the technology. And I can show you that. So we shouldn't be fighting about that anymore. Yeah, you gotta. You know, you may want to worry.

Is it a big sample? Is it a little sample? You know that kind of stuff? But otherwise no, it really is a better deal. And that's really what we should be focused on.

Well, you know, you can use a combination of the 2. But responding to the first question, the answer is, if Dr. Jeremy Pickens will join me again. There will be growth. I mean. Jeremy and I were working on. You can't imagine the things we were working on. We were working on image TAR. We were working on multimodal Tar. That would look not only at the text, but at the date, at the sender, at the recipient. to start looking at the factors that would influence the model and influence the efficiency of the model. So yeah, you can make it better. There's absolutely no doubt when Jeremy and I worked together. We were constantly making improvements all the way along, anyway. And now we've got new techniques that we could look at.

And and you know, when you when you move into the world of where technology is today, AI, for example. there's still room for TAR is in my world. But yeah, TAR, even putting AI gen AI aside. TAR can improve without a doubt, can do more.

Look, let me put it to you this way. Every every 45 documents ish that Gen. AI finds saves you an hour's worth of people's time. Right? Is it gonna save you every dime of people's time? I don't know. I doubt it. I expect that what we're gonna end up doing is having some measure of Gen. AI tied with review. I expect that's what's going to happen, at least until we really learn how to refine prompts, although I don't know I don't know where that's going to go. I'm in the process of doing a lot of testing on this to see any number of different things and how they will impact it. I just. I look at Gen. AI, and I believe that it will benefit the process it is, though, Daniel, as you said, it's essentially a TAR one in sheep's clothing.

right? It is, generally speaking, the same thing. You are building a model. It's a different model to approach the problem from a different way. But it is a. It is an effort to build a model. It happens to be a big neural net model, right? That none of us are going to understand. But it's a model, nonetheless. And and you're trying to build that model to reflect a way or to find a way to grab the documents that you need. We'll get there. We'll get there. It's going to take time. It's not today. I mean, I was, I was telling somebody earlier. I read an article where? Where someone said, Oh, the client has saved 70% of its costs using Gen. AI, and you go to.

I started reading it. And the step one apply keywords. And I'm like, Well, that's that's just not fair. Okay, no, you can't take credit for that. So yeah, it'll get there. It's going to take a while. But Brandon, back to some of the stuff that we talked about. You've got people on the other side you're gonna have to deal with on that.

We we have. We have done a good bit to get people to understand validation. I don't think it's a hundred percent of the way, but more than half the people that would have argued against me before were arguing on the issue of validation and happy with it today. So we'll be able to get, I think, using validation. We've already moved much further down the line than we used to be, because you now know what you're seeing in terms of validation, what types of validation approaches, what kinds of numbers, what kinds of analysis need to be done? So I think that's, I think that's going to improve on the acceptance of Gen. AI. Just my, my guess is that will help. We're still going to have folks who are skeptical about it.

And look, there is a reason for it. I was telling somebody this the other day, I said, here's the problem, and you need to understand it from the perspective of the requesting party. because I know. Look, I'm I'm not suggesting that I want people to do this. I just know they will. They are going to want to see my prompts, and I will tell you why this and this isn't rocket science, because you know what my prompt says my prompt says, go find me 80% of the responsive documents, but make sure you never include a hot one.

And if that's my, if that's my prompt, you've got 80%. Yay. And now look, I don't. I'm not accusing folks of doing that I'm just reflecting on the fact that those who have a concern about validation have that concern. That stuff they really want is going to be missed. We'll get there, we'll get there. It it ain't tomorrow.

Well, it's it's going to. It's going to head in the direction of focusing on AI generally, right? And the theory that AI is going to be able to identify documents. You, you run the documents through the AI. It gives you the stuff that's responsive. We're going to have people focused on. Prompting. Right? We're going to move in the direction in my world. We move in the direction of building AI together, Gen. AI, together with TAR. so that TAR is always suggesting the next most likely responsive document. But your AI is doing the decision making right. So at the end of the day you have actually reviewed the least number of documents you possibly can, because you get down to the bottom. And you see there's nothing left.

The other thing we're going to do. The other thing we're going to do. There are 2 things that I want to see happen. One is, let's build in the Qc. Loop into that process, so that the feedback that the AI gets is the same feedback that a lawyer would have gotten when they were doing the review memo. So now we're constantly updating the prompt to make it much better at finding those documents. So let's figure out how to build in that Qc loop. And the other thing I want to do. Sorry to the law firms out there who want to spend all their time crafting review memos. But one of the tests that I'm running is, how can I take the complaint? The answer the Rfps and then go to the data set, and my first prompt is going to be.

Hey? Here's a complaint. Give me the top 10 documents that support this and the top 10 that refute it, and then I'm going to go. Here's all my stuff, Mr. AI. Craft me the right prompt to go get these documents. and I just saved. I just saved you a hundred hours of attorney time. Right? So that's where I think we need to be looking at every component of this process and finding out how we optimize it for all of the tools that we have available. and make sure that we're getting. We're squeezing every last ounce out of it to keep the cost down and bring the efficiency up. And heck! Think about this. We may actually get to try cases again. Oh, my gosh! I mean that was my biggest problem.

I like to try cases. I hate discovery.

You just take my documents, you can have them. Let's go try the case.

Well, I mean it really. For me it's pretty simple I've been to. I've been to every conference I've been to just all over the place to different ones. And my own view is that NeLI provides the best substantive material to people who are practicing in this area, and need to know what to do, where to go and how to do it, what tools are available, what you can expect, what you're going to see, the way that process is put, and it's so nice because it's a. It's a small kind of auditorium setting that everybody sits down. Questions. Get asked questions, get answered, and you don't have to go look at different places to go different sessions. We all get to learn together, and that has the benefit, not only of the speaker.

and what that speaker can give to the audience, but what the audience can give back to that speaker, and what the entire group can take away. And I am telling you, in my view, it provides the best substantive material of any conference going today in from my perspective.

Oh, man, you are! You are absolutely pleasure. It's great to be with. Be with both of you guys and have the opportunity to do this. I just hope that I don't send people screaming into the hills. And they never listen to another. NeLI, podcast although they don't have to listen to Judge Peck's.

It's a ringing endorsement. Give him my best. Give him my best. Will do all right, guys. Thank you.