Guardrails and Gaps: E-Discovery, AI, and Compliance in Legal Tech

From year one in RelativityFest, David Horrigan put together the e Discovery State of the Union panel in which we gather together theoretical pontiffs and folks who love to pontificate on stage. In this case, it was twelve different experts with a panel of four judges off to the side. I call it eDiscovery speed dating because we're asked to come up with our top topic for twenty twenty five in eDiscovery, and we have three minutes to deliver an elevator pitch to it, and then the judges and the audience judge which one they like the best.

It is fast, furious, and fun.

Well, the secret is that everybody wins.

I was lucky and unlucky enough to start in this industry in the police. So I was a CSI originally in Houston.

Back then, I was intending to go into it to do ballistics because that was my degree and where I wanted to go. But as the youngest person they'd ever hired in the crime lab, I got stuck with the computers because they thought I would understand them. So I got to do the pulling texts off of pagers and doing all that. They got me started into this industry.

Since then, I've worked in law firms and corporations and designed software and done consulting.

And thirty five years later, still having fun with it, enjoying it. This year has been remarkable for me because after about a ten year hiatus of doing strategic only consulting, I've gone back to cases again, and I love it.

So I think it was twenty seventeen, our market analyst business was changing. We looked at the market, decided that as a team that that was not really going to progress the way we wanted to. I went back to consulting, but without having a full time research budget, I couldn't track the whole market. And I looked at all of my big global clients that I'd worked with and said every single one of them is on Microsoft and every single one of them is moving to three sixty five in the cloud.

So I bet on Microsoft back then.

And what I've done is focus all my research on it because the reality is when it comes to large corporate litigants, they're pretty much all on Microsoft three sixty five. That's where all of our data that's not on phones resides and we have to be able to deal with it. There's not a choice. You can say you don't like purview, you don't like what you're getting out of it, doesn't matter. That's where it lives.

We have to, as practitioners, understand the infrastructure of how it is stored and what that means on what we retrieve. We have to understand the capabilities of the system and that it is constantly evolving, which means we have to be better about trust but verify. Figuring out what are you really getting back for your data, because I will tell you that every client tenant on Microsoft three sixty five is not the same.

Your data's not the same. Your usage amongst your engineers especially is not the same.

And so you really have to get in there and run some plurates. It's like any other system. Validation testing, acceptance testing.

Understanding with known data sets, broad searches, how do your terms work, compare them against data sets you've already got in a review platform.

So you've already pulled full mailboxes, full OneDrives, you've put it into a platform for this litigation, go back and rerun your searches against each other.

The answer to that is yes, but, which is the classic consultant answer.

As attorneys don't have to understand the infrastructure, but they have to understand the limitations and the implications on retrieval and preservation.

So they don't have to understand the technical side of it, but they have to understand how does that impact the criteria and the ability to scope what you're looking for. That's where it's important. So the technologists on the ground floor, whether it is a Lit support manager, whether it is your preferred vendor who is a managed service, has a managed service contractor, whoever your expert is that's running that validation testing, the output of that testing should be a simplified document and report that a nontechnical attorney can read and have translated for them if need be, that says, our people are using this part of Microsoft and we can retrieve parts of it, but we can't get this.

You know, for instance, loop components, we can't reattach things, or the latest version of the Microsoft automatic copilot summaries that are generated every time you open a Word document for somebody who's licensed that way, we can't retrieve them. You can see them as a user, but we can't get it back.

That's the type of area where it needs to be given to them when it's contextually relevant to a matter. You need those kind of declaration docs, especially for your retained counsel Because retained counsels coming in, don't have a background.

So it's part of what I would call kind of building the Bible for corporate eDiscovery, which says, pull together your information about you can call it a data map, you can call it just kind of an overall broad view of how does this company do business, how do their people store data, what kinds of data, and where is it? Microsoft happens to be the biggest one for standard documents and more formalized communications, but it's not the only one.

Yeah. But I'm drawing a line here because what I'm finding out in the world of large there are regulated industries, especially financial service companies. There's folks I've worked with that have extensive in house teams. That's your technical expert.

Most, even global corporations, don't have a permanent on staff expert and they don't need it. They need expertise and they get that expertise most of the time through their preferred providers, whether it's on cases specifically or whether they have an independent relationship with that provider. That's it's something that we saw in the very beginning because I'll be blunt. When Eric and I were long past early days of being eDiscovery managers, directors for companies, law firms.

The salary was really good back then. They paid a lot. They don't anymore. The market is evolved. It's matured.

When you have someone who's an expert whose billable rates can be very high, you need to use them appropriately for that analysis when you need them, but you don't need to be paying them day in and out for that kind of level. That's why it is the expertise has shifted to the preferred providers because they can come in on cases specifically where you need it, they can run things like acceptance testing, they can give you their advice, they're on call, but you're not paying for them twenty four seven all the time. So that's why I put that fourth leg on the table there because you really need those all together and the role that I'm finding myself this year play is to play cat herder around all of them because I'm not always the technical expert.

There are specialty areas that I don't cover, you know, and I'll need folks in there. Mobile phones, as much as I've done them in the past, have evolved so quickly that I don't call myself an expert on them anymore. So you've always got to have someone who's gonna be more than one technical expert. I can cover Microsoft.

I can cover areas that are really good at, but if I'm looking at a very, very expensive review with terabyte upon terabyte of data that actually has to be gone through, then I'm gonna wanna have a specialist in there who is sharp on the chosen tool, whether it's AI, TAR, otherwise, we're going to apply to back me up. I'm not gonna get in there and put my hands onto a tool that I have not extensively tested and worked with.

On all of this, you've always got you always have two levels of what I call documentation defensibility. There's the internal documentation that says everything gets written down, every little bit of it, because you might not be there when it needs to be defended.

You have to have it, it's got to be documented somewhere clearly. So all those prompts, all that information needs to be somewhere locked down on a history level. And then there is the disclosure side of it, in which we did this with Tarr. In the early days, we said, hey, I'm going to use some advanced training on this, we're going to reduce the documents.

Honestly, in the end, it's not that much different than search terms. The only difference is that counsel now wants to agree back and forth between sets of search terms. I disagree with that a lot of times because, frankly, I don't think either side of the council until we touch the data understands the terms at all. And I've walked into too many clients who said, hey, come look at this case for me and found out they'd already agreed on search terms before they collected.

And I'm like, what are you doing? Have you tested them yet? How are they defensible? So the validation of process and the ability to provide an attestation to the court should be something that covers the overall process.

It doesn't need to get into details unless there's a conflict or an issue.

I think it's very important to disclose your quality control.

How do you know the process worked? I don't think it's as important to disclose your prompts or your technique that is your unique capability, I think it's very important to come back and say, we did this overall approach. This is how we tackled this giant couple terabytes worth of data.

And at the back end of it, here are the statistics for the quality control checks for the recall levels, for illusion that we're finding. Here's the area where we've done spot checks and we've, you know, we've done statistical random sampling and we have we've we've come we are comfortable that there's a reasonableness to process and we don't have systematic evading of topics that are relevant across this dataset. That's what's important for divincibility, is not, you know, going up and publishing every single way that you're approaching a problem.

Instead, it's how do I know what's right?

Well, there's a couple of thoughts on that. One, and yes, there are disproportionate positions where we've got plaintiffs who don't have all the data, but when it comes to most corporate to corporate litigation, I constantly have to remind counsel that just because you think that we're the only people who own copies of this, you're wrong. If it's flew back and forth, I can't tell you how many times I've had folks say, well, you know, we looked at this email string and it's fine, and I've been like, yeah, all that was ported out over here. That person walked off with it. So the other side is going to have data on it most of the time.

I believe that it's tackling this is always tricky about the bias that goes into things.

I think that that's where old school recall testing, old school sampling, this is where you have to be able to do it. It's it doesn't have to be high volume, but it does have to have the randomness and the ability to catch it because it was George Socha who said, believe it was Monterey Bay first year. He was in the room and he said, it's not getting the ninety five percent that I know about. It's that one percent that I don't know about, that I'm worried about. Give George credit on that one. And it's true, because search terms are biased. They're biased on what we know, and that's where I go back to that first statement of you can't create scope without contact with the data.

Cause until you have contact with the data, you don't really understand the parameters under which, you know, the date range, the people, all the rest All the stuff that might be nuanced.

Well, a good one would be for the eDiscovery State of the Union. I threw out three different topics. The team picked one of them that I could take and run with, which was the deregulation issue. So what are we doing now with effectively the evaporation of regulatory enforcement in the market right now?

And although I had good information about that from cases and from attorneys who'd come back to me and talked about their interactions with regulators and having called a couple peers who were also heavy in the regulatory space and gotten confirmation that they were seeing the same thing, What I didn't have at that state was statistics. Hop into Copilot, run a bunch of queries on the public and the first thing I walked in there and asked is give me a list of every regulatory agency and what the reductions in staff and the reversals in rules have been since the beginning of the year in twenty twenty five.

It dumps it down and I walk it through the next and walk it through the next, just kind of playing with it back and forth, output that into an Excel spreadsheet so that I can actually manipulate it and format it right. There's my slide that is giving background and from that I was able to turn around and generate a list of citations that I went through and opened each one of them and checked and sure enough, Copilot had pulled in my twenty twenty four survey that did not align because it was not within the time range that I want on there. You know, you have to quality control these things, but that would have taken me two or three hours of traditional research.

And I was able to do it in about twenty minutes and ship David Horrigan his slides that he wanted and have real content for a CLE backup.

So I'm not just up there talking about what I've encountered directly with counsel and regulators and cases and peers, I'm actually talking about what are the statistics? Ninety seven percent reduction in regulatory enforcement penalties in twenty twenty five.

Right? So let's talk about speed and let's talk about safeguards. Yeah. Okay? Speed, it's what we see.

It's just hard to do because in eDiscovery litigation support traditionally, we are there to serve lawyers. They are there to give us the legal direction. They are there to tell us what their goals are. We provide the technology to make it happen. I've had this discussion with a provider in the last week in which I was sitting down and saying to their project manager, there is no no in eDiscovery. There is how long, how much, how hard. It's very hard to protect counsel when they are in charge.

But, and when it comes to the practice of law and to their motions and their filings, I'm not even going to go there. It's not my area. What we can protect them from is overstatement and overly certainty. I do this all the time because for many of my clients I am their thirty b six for the company.

They brought me in once upon a time to help them with their process, to evaluate, build that data map, build that Bible, and they've got me on retainer and once every six months, sometimes for some of them once a year, I go in and we do a level set, how are we doing, what new technologies have come in. The point of a lot of that is to go back and to be able to work with counsel when they say, hey, I need you to write an affidavit about this retrieval and, you know, we searched everything, didn't we? And I'm like caveats. We're gonna state what we could search.

We're gonna state how reasonable it was and what our criteria, what we did, we're not gonna turn around and say that we've researched everything in these cloud systems. And I'm gonna loop all the way back a little bit to an early question you said about the kind of the gaps and the other things. There's a fundamental architecture within Microsoft's system called the substrate process.

Everyone thinks, oh, I'm searching loop. I'm searching teams. I'm searching all these different applications. And the answer to that is you're actually not on most of these. So stream for the videos and all this stuff.

What Microsoft did when they tried to create an architecture that would support universal search, global search of all this data, is they realized that maintaining all those separate indexes was very difficult, that every system maintained different architecture, different databases that held all the metadata and all the rest of it. And so instead of trying to concatenate and bring together all those different data sources, they created a system in which essentially record copies of the root information was stored in only two locations.

So Microsoft's searching only two different pools, mailboxes and essentially SharePoint drives, which OneDrive is a SharePoint version.

That's the only thing that's indexed, and that's where the record copies live even though the live copies may be existing in a various application, whether you're talking about planner or you're talking about, you know, all these different apps, the thousand little icons that we know that Microsoft three sixty five gives us advantage. The way that they make your search work is they put the compliance copies and they bury it in hidden folders in your mailbox or they bury it down in folders on your OneDrive. And because that is indexed live, that's where the copy is. And the substrate service is service behind the scenes that looks at, oh, I've got a new planner task.

Let me grab that task and let me route it down and tuck it into your mailbox so that search wise you can now find it with a pointer. It's there. That's important because it comes down to the gaps. When Microsoft develops brand new services, brand new document types like the fluid types that are behind loop, They grow the new application first.

They look for it. Is it getting adopted? Is it getting popular? Do people want it?

And then they connect it to the substrate system. So there are time periods in which we have to watch for is where's Microsoft innovating? Coming back, the main area that I watch for is overpromising. The way I protect my attorneys from getting us in trouble is to fight the good fight.

Sometimes they want to oversimplify the statements. I can draft an affidavit. I've learned not to let them draft them for me, but you go back and forth like, can we just say you got everything?

You know, I'm sorry, but I'm a consultant. It depends.

So that's really where we can do the best job. We can't protect them on they're gonna use generative AI on their filings and the rest of the stuff, I'm sorry.

That's outside of scope at this point for our world and what we do. We can protect them from over promising, overstating. I mean, they're right, when they're signing off on a production, I have to go through and review that very carefully because we have not looked at every single document and every single custodian's OneDrive because we use search terms. We have not looked at every single email. We haven't dug down. There are parts of that mailbox that are very difficult to crack open and pull data out of.

You're welcome. Thanks for having me on, and it's been a great fest, and it's been really enjoyable.