Martha Louks on Modern Data, GenAI Review and Cutting Through the Hype

Thank you so much for having me. I'm excited to be here.

Ooh, that's an interesting question. Can I say all of them have done both?

In different ways?

So, okay, so going from paper to databases, the big innovation there was we can all work in the same place and share work product in one spot. That was the selling point back then for me to attorneys was, you know, if we work on paper, then everybody is siloed in the specific thing that they're looking at. If we put it into a database, we can share work together and collaborate. And so there's more efficiencies to be found that way. So I think that kind of brought efficiency and standard practice into a little bit more of a more visible way than it was. There were, of course, standard practices with paper, but not in the way that they're it was more regimented when we had to do it in a database because everybody had to be clicking the exact same buttons.

That's what we had was one responsive button, at least you hope there's only one responsive button. But TAR, TAR was interesting when we started doing machine learning. I guess, you know, even just the advent of ESI was sort of interesting. The fact that you could deduplicate documents was, to me at the time, extremely exciting. I had reviewed a lot of dupes. And so it was exciting to me that the computer could do anything. And I remember talking to people about the rigor of hash values. There was a lot of, a lot of, well, I don't know if it was skepticism exactly, but lawyers were doing their due diligence as to the technology. Like, how robust is the deduplication? How do we know for sure that a duplicate is in fact a duplicate?

Well, it is. Now we don't even think about it. We just do it. But back then, that practice was asked about and had to be defended. And the same thing happened with technology-assisted review when we started doing machine learning with it, whether it was, you know, simple passive learning or continuous active learning. How do we validate it? How do we know the results are good? There were a lot of external studies that were done that were cited to, and it took a long time for attorneys to become comfortable with using the technology. So that one, that one took quite a while for, I think, you know, there was court acceptance of the use of the tool, government acceptance of the use of the tool. And now that's all, that bar has already been cleared.

I think there's faster adoption with GenAI tools. You know, there's a little bit of familiarity with the process. It's a new tool, but the same rigor is used in validating the results from these new applications. So, you know, this one, there's a little bit more explanation into, you know, the details of process, but there's less skepticism about the potential benefits of applying a new technology to workflow this time around.

This, you know, it is interesting. This, I think so. I would say so. I see a lot of different, I see a lot of different job titles out there. So there is certainly appetite in the market for technology professionals or hybrid legal technologist roles. Folks who have, whether they're attorneys or not, they at least have experience working on legal matters. And there's a lot of growth, it seems, in knowledge management and innovation positions, at least at law firms. And in-house, it seems like there's a growth of AI-centered roles recently. Now, I'm interested to see how sticky these roles are and, you know, what value these organizations are finding from the different positions that they're developing. So it's, there's a faster adoption. It feels like there's this almost like a race right now in a way that there didn't seem to be in the mid-2010s, mid-2000s.

Comparatively speaking, there's a little bit more big tech hype around GenAI that I think has flowed into the legal technology space that didn't necessarily happen the same way with machine learning.

Yeah, they are.

We're just going to keep rocking the document object, it seems to me. I've been talking about this for years now, that it's data, not documents. I think I started on this during COVID where I was like, it's the document, is it even real? You know, I mean, a little bit from paper days, sure, but even email isn't really a document. It's a database entry and it's segments of communication that could be parsed into smaller data pieces and then strung together. And yeah, it's a real challenge in our systems that we are kind of locked into this system of documents. It's also interesting that we still produce largely in TIFF image. We make fake paper, you know, and I mean, it is what it is. I guess I don't know if you have to redact, I totally understand that.

And you do want maybe a page number, particularly when you're citing to something. But largely speaking, you know, it increases the hosting size. It's a lot of files. It's a lot of stuff to manage. And it's like shoehorning modern data into the format of yesteryear. It is kind of a curious thing. I mean, there are some reasons for it. I do, I see them, but I do question its continued utility to us. What I see as the big challenge is that when we're focused on these so-called documents, it's really just a predetermined slice of data. We are lacking the ability to track activity and analyze activity across sources and put together a timeline. So if I, for example, let's say I've got, you know, Slack chats or now I guess people could be grabbing, what's kind of interesting, you're chatting with AI bots all day, Claude or ChatGPT or whatever.

Are we going to put those into 24-hour slices too? Does that make sense? I don't know, maybe. But in any event, if I put all 24 hours into one document, my date for that document is the first date and time of the first communication of that day. And everything else gets collapsed. And it's not really something that I could analyze. So if a person is chatting all day and then at three in the afternoon, that chat inspires them to, you know, go into another platform and do some activity and talk to another person about a topic that's relevant to the case, that's going to be harder to piece together. It's just going to take more time. So I'd love to see innovation in platform design around this.

It's something I would like to see. I don't know though whether it's the most important thing for many attorneys in practice in the way that the way I think about data does not always match up with how an attorney might think about data in a litigation, you know? And a lot of times I would go maybe down a path of deep data analysis, but it's simpler sometimes to just ask a custodian. You could just ask them in a deposition. It's another way of doing discovery.

There's a little bit more trust, but what things are we getting wrong this time or perceiving wrong? The hype is frustrating, to say the least, to me. Yeah, there's a lot of misconceptions about AI being able to do a lot more than it can actually do in practice. You have to check everything. With eDiscovery, it's kind of okay if it doesn't get it right 100% when we're doing like a big doc review, we've built in margin of error. So if it gets it wrong 3, 5% of the time, you know, okay, I'm not expecting a perfect result. But if you're using it to draft a brief and then you're submitting that to the court, that's a whole different ball of wax. I mean, that is much more concerning and there's a higher standard that needs to be applied to that work product.

So I think there's a difference between using AI on a high volume task versus precision work, you know, and you have to verify all of the output. So the question to me, especially for precision work, is just the time and cost of verifying the output actually make the cost savings negative or are you still getting at least some return on your investment in the use of the tool? I think there's just, there's a lot of hype. The marketing vibe is very, it's very simplistic, you know, that AI can do in minutes what used to take hours. Well, maybe. I mean, I don't know how many minutes are we talking about? How many hours was it? How much are you going to have to verify? There's just, the devil is always in the details and a lot of the hype, it smooths over all details.

And a lot of the tools, they have huge promises, but they're very thin on how they present, at least in marketing materials. They're thin on talking about the concrete ways that they work. I would love to be able to just go on a website and be able to read and understand like, oh, okay, it has X, Y, and Z function. It does these things. And, you know, without having to necessarily talk to somebody and go through a sales cycle to understand the basics of a product. Right. So yeah, there's a lot of, and there's a lot of fear-mongering too this time around. There was before, but it's worse this time. A lot of like, oh, it's going to replace the lawyers. It's going to do this.

And I feel like that's another sales tactic in a way, you know?

Well, so with a doc review, I'm still going to have to validate it. And using GenAI is still technology-assisted review. So I'm using it at scale. I'm still going to have to do an illusion test or some kind of sampling. And at some level, you do want to know what's in your documents that you're producing, at least. So you can chat with the system. You can learn some stuff. I think the funnest application that I see, the easiest application that I see is when you have your hot docs, it can extract facts and help you create a timeline. That's so much busy work that people would spend time typing in the facts or copying and pasting from the document. You're going to still need to confirm that the fact is correct, but theoretically you're looking at your timeline and thinking about the results in light of the actual evidence.

So that would come up as part of your QC process. But to me, that's like, I think that takes a lot of manual work away. So that's one area that's very useful. And I think it's great for, you know, early case assessment or an investigation where you don't really know much. Like just run a prompt, throw it in, see what comes up, and then use it to learn a little bit about your documents. These were all techniques that we've been talking about for years and years and years. So it all just depends on the appetite of the team to kind of get their hands dirty and start digging through the data a little bit. You're going to still have to think. You need to use your brain.

That's never going to go away.

I don't see why it would.

It shouldn't.

From the statistical standpoint, no. What's interesting to me though, we have studies that show that computer-assisted review is more effective and higher quality than a large review team. And it's basically because the computer is consistent, super unfailingly consistent in a way that human beings never could be, right? So if you scale that across a large set of documents, there you go. You've got big consistent results. I have never had an opposing party interrogate a manual review and say, produce quality statistics to me. Prove how good your review is. Prove how good your search terms are. We trust people more than technology, even when we have evidence that says we should trust the technology more. So that is really interesting to me. And I wonder if that's still going to continue to be the case, but there's a trend of, and I don't know if it's even a trend, it just is the way we interrogate the results of computer-assisted results far more than we do human results.

And yet the results are the results.

What was the question again? Sorry.

Yeah. I see a couple barriers to it. Why don't we? In practice, it's so it's a good thing that there's case law that discusses technology-assisted review, the acceptance of it by the courts, validation, and, you know, having a defensible process and all of that. However, I think that in the, there's a consistency in that the case law wants the parties to collaborate with each other and in an adversarial litigation, when you start talking about a technology-assisted review protocol, it's like fodder for another fight. And so a lot of times I'll see parties say, no, no, no, we're just going to do search terms. I don't want to have to have this argument about it. Now, interestingly, there isn't an obligation to disclose your use of the technology, at least in, you know, civil litigation.

There's Sedona Principle 6 where the producing party is in the best position to determine the means and method of production, something to that effect. So I say, go with it and then focus your reviewers on the meatier substantive analysis. Use the AI to lop off the non-responsive stuff. Don't waste the time reviewing that anyways. It's not going to add value to your case substantively. And then focus the review team on the issues-related things. You know, we've got classifiers for that. You can use GenAI for that. You could use a chatbot. You can use clusters. You can even use search terms. You can do all the things and find the interesting documents that actually add value to the litigation process. But I think there's a sense that you have to have agreement on all of these details.

And in an adversarial case, that's kind of a heavy lift, you know? I mean, who wants to burn political capital on that issue when you are going to need to use it for something else in the case? But I think say less, do more, say less.

I think, yeah. And I, you know, there may be some cases where there's unnecessary review that's already happening. And maybe that kind of goes away a little bit, but there is still a need for review. You're still going to have to know what is in the stuff that you're producing. You have to know if it's helpful to your case, if it's harmful to your case. You're still going to have to assess documents for privilege. I don't think the standards are disappearing. The privilege doctrine is, you know, hundreds of years old. So we're still going to care about privilege. Logs are not going away. Privileged logs are still going to be a factor. Even if there's a draft description, someone's still going to have to confirm that's accurate and worthy of including on the log.

And then there's other new functions, like one of my favorite new things with GenAI, it's new-ish, is using prompts to structurize documents. So if you've got, let's say, 100 PDFs, let's say they're letters, I can use a series of prompts and say, what is the date of the letter? What is the provision in this paragraph? Who is the recipient? And it can pull that information out and put it into a table. That's a type of project that can be very useful for a number of different use cases. And that still needs to be reviewed and confirmed by legal eyes. So the nature, that's something kind of new. It's almost like bibliographic coding in a way, in a way. So yeah, there's always going to be a need for humans because this stuff isn't perfect.

And if you are going to rely on the output for anything truly substantive in a case, then you do want to have some assurance that what you're relying on has been confirmed.

That's a really interesting question. A little bit of a shell game maybe, right? So I guess the way I would do it is calculate my cost. I look at the total project cost. So there's technology fees, processing, analytics, you know, whatever it is that you have a line item for. If I'm reviewing less and using more technology, then I have to consider, is there an overall savings for that project? One thing we haven't talked about though, the volume of information is still going up. That is not stopping. And the nature of the information is changing too. I don't know if you guys are seeing it, but there's a lot more video, a lot of recordings, a lot, you know, I mean, all these different chatbots.

There's just more sources for more information. So even if we review less as a percentage of the total, that doesn't mean that the total amount of stuff that has to get looked at is actually less. Do you know what I mean? Yeah. Yeah. So I may have to spend more in technology to review slightly less or the same amount, the nature of the work. It's hard to see it disappearing. It's hard to see it disappearing, but yeah, I would, you'd have to really think through your workflow. I like to combine the tools. There's nothing wrong with old-fashioned classification either, you know, like technology-assisted review. Do that on a data set, get ready your non-responsives. Use GenAI on some, you know, targeted subset, perhaps, if that's a cost concern.

And then, you know, use the tools that make the most sense for the task at hand. And then consider budget on every level. But yeah, I don't know. I guess it depends on how the users approach their budgeting and workflow. Yeah. Yeah.

Be flexible. You just got to be super flexible and have strong fundamentals. Because even with the new technology, as we've been talking about, this is a theme that's come up in this conversation, the fundamentals remain the same. You know, I'm going to approach a project today with the same rigor that I approached a project in 2012, actually probably even more. But we have to be super flexible. Every six months, we're encountering some weird data that, like, it came out in a different format now than it did six months ago. So you just need to be strong on understanding what it is that you're looking at so that you can kind of massage the data into something that makes sense. Being flexible to understand and staying up to date on new tools, like what are people using to get work done?

Because that's where the information will live. And so there's going to have to be folks, it's about be flexible, process-minded, and adhering to the core principles of good work and staying curious on things and skeptical. I think a healthy dose of skepticism is really useful in this industry.

Thanks. This has been a fun conversation.