John Benson on AI, Security and Protecting Client Confidentiality

Thank you.

off risk? That's a fascinating question because with so much talk about the validation off the different tools and whether we're talking about discovery or anything like that, it can be difficult too be… mindful off there being so many other ways too validate things in the real world versus validating things in kind off a technical vacuum, right? So take, for example, let's say you're going too due some legal research on a fairly well-known-too-you topic. You're going too get some results. You're probably going too know those cases. I had some cases pulled back for me by a system last night, and it was some cases I hadn't heard since law school, but they were fundamental concepts off a certain area off law. And I was like, yep, that's the right case.

Yep, that's the right case. And as far as I know, I haven't looked at the news tooday, that case is hopefully still law. So the areas off experimentation have become. different. more available too people in a way that they can use it in their cases and validate them on their own stuff as they go and as they incorporate it intoo their workflow. So, yes. It's very neat. That's a tough one. It is.

Yes.

One off the things you've got too due for that is too simply start by defining the risk that we're trying too control. Because if you define the risk, then you can put the controls around the risk, both individually and as an organization. And so when all off General AI came out. One off the biggest, biggest and most broad concerns, certainly one off the topics that we've covered over and over again and we kind off continue too hear, even still at, talked about, is concerns over training data and confidentiality. And this isn't too invalidate any off those concerns, but at this point, it's fairly easy too say, this is a concern off ours. We want too put a control over that. We only want too use systems that we know have controls over that.

And luckily, that's become a big enough issue that users now have more control over that kind off thing. There was a time when a lot off the Gen AI stuff started coming out and a lot off the kind off storylines that have persisted about Gen AI since and reliability, security and everything like this. That were just so early that if you were a ChatGPT customer, even if you were paying the money, they were taking your data and they were going too continue too train on that data. They did not give you a choice. The only way you could due that safely was too go through a third party that was going too use an API, and that was safe. But that was going too be costly and made it very difficult for people too experiment.

Now that we understand that, that's a simple control. So if you're an individual or business, you can identify, hey, let's go find that box and check that box, at least for now, and then feel better about some off the information they're putting in there. And then even more so, beyond just defining confidentiality, we also need too kind off define what's the probability off this happening. And that is going too be difficult, but I think we have too be reasonable. Right. There is, you know, you see research papers and a lot off stuff that comes out and gets a lot off play and certain early on that you could give certain language models or maybe small language models and, you know, enough garbage data and they would just vomit back training data verbatim.

Right. So that's a big concern. That's a big concern. But at the same time, look at the size off the parameters and size off the training data that's going intoo the large language models that anybody right now is probably responsibly using that would be based in the cloud. It would be putting your data out their and available too other people. That risk may not be as high as the number off news articles indicates. Yeah.

I think it is, but we also can't ignore that so many off the concerns are parallel too cloud storage, right? So take the LLM out off the equation, especially if you're talking about the web-based tools that most people use. Once you get past the fact that you are bouncing it offf off an API and it's not hoovering it offf for future use when it hits the API, all that data is sitting in the cloud. So is that cloud data controlled based on your firm's policies? Because it's cloud data, just like it would be in Gmail. Those are the kinds off controls that we go back too. And a lot off it is kind off a return too those kind off fundamentals, right? Yes, AI is a new wrinkle too all off this.

But the analysis off some off these things is still fairly simple.

Boy, I didn't expect to come and talk about compliance versus security, but boy, here we are.

You're exactly right, because one answer would be to say, hey, find people that have good policies and check good boxes and things like this. But some of those boxes for compliance may not always match up the current risk model, right? Right. So some of that may relate to a lot of it's going to be the different choices that people are going to be making right in their in their use cases. So we shouldn't treat everything like it's all everything. You know, if you're looking for a search engine that you're going to hit up while you're driving down the road in the car, you know, perplexity. Yeah, OK, it's probably fine. But, you know, are you going to trust perplexity necessarily with client data? You probably want to do a little bit more due diligence and due diligence and due diligence.

And then. Think about. Do I really trust all of this, too? Because in the cloud area, it's not that hard to be able to say that you're SOC 2 compliant without really doing a whole heck of a lot of work other than just choosing the right data center. And I choose the models or hosts, you know, I try to always stick to the first parties as much as I possibly can, you know, and even, frankly, for other kinds of resellers, let's say, you know, so for example, we can talk technical a little bit here, right? So let's say you just want to go out and you want to spend money on experimentation. You can either bang your head up against the wall and sign up for 40 different accounts in the cloud and get an API key for every model.

Or you could just go to OpenRouter and give them $40 and go play until $40 is gone doing anything from making voice to building agents to anything like that. Right. But. It's going through a reseller. It's going through a router. Yeah, they've got a privacy policy. Yeah, I've read it. Yeah, and if I read it, if I read it, yeah, I could probably trust it, but I don't. So from my experimentation, I'm not going to put real client data through there because I don't know necessarily where it's going to go. But, you know, stick to some of those first party providers, stick to countries and companies that are big enough to be trusted as much as any big company can. Right. So there was the, boy, we've had the DeepSeek thing happen.

Right.

Yeah. So DeepSeek is a large language model that was developed by a lab from China and it was released and it came out on the App Store and it was super effective and remains super effective. And when these things come out. So it's easy to be blinded by the marketing and not be able to see the difference between the large language model and the products that are being sold tied to the language model and that therefore fund future development of the language model. So DeepSeek comes out and is really good. DeepSeek R1 comes out and DeepSeek R1 does this thing called thinking. And it looks magical to a lot of people because it does this stuff called chain of thought reasoning. And chain of thought reasoning was something that to, I would say, most people just browsing the app store for apps for AI had never even seen such a thing.

And DeepSeek released the R1 model and they also released an app. And that app was hosted by mainland China. So it was incredibly controversial because data was going there, first and foremost. And that privacy policy was not very favorable to anybody. That was a dangerous thing to do, depending on what you're putting into it. And then there are also a lot of concerns over the behavior of that kind of model. There are a lot of concerns over disinformation and the fact that it's coming from a Chinese lab, just first and foremost. And frankly, DeepSeek R1, if somebody wants to learn about language models, I think R1, running locally on a computer, is one of the best things that a person can play with, because of the opportunities that it affords and how verbose it can be.

But you don't go to the Chinese government and say, hey, can I use your model on your servers? You know, go find somewhere else to do it. And you can do that via some other places or bring it local and other things like this. And, yeah, so much is so technical. And I would say when I started doing this 18 months ago, damn near 24 months ago now, you really had to go and scour for the stuff, for the resources, for ways to do this. Because now it's so much easier to go out and a person with ChatGPT can just check that box. And if you want to go to any number of services that can let you try the same prompt on Gemini and the same prompt on this and the same prompt on that, they're all out there.

It's a lot easier. You don't have to make friends with the developers like I did of those tools to be like, hey, help me out. If you add this, it's going to make my project of copying and pasting a lot easier.

That's a good question. When I started a lot of that research, because I started essentially a benchmark called Gunner Bench that is practically based. Because there… I know… I knew that evaluation of things like eDiscovery tools was going to be a well-worn path by people that were getting money to do that. But I was interested in saying, okay, well, I started using these models, and I've just started throwing random stuff at it. And I know that some of them are expensive, some of them are hard to get, but a lot of these are incredibly capable. And so how do I look at this to say, A, what is the best, but what also gets me by? And I started looking at that earlier on from the perspective of drafting.

Because I think legal drafting, anything that's going to involve so much complicated synthesis, you're always going to want something that's really top tier to be taking care of that. But as we get more options for models and as the technology takes all of these processes, and the eDiscovery is the one that we're most familiar with, right? You know, eDiscovery right now, yeah, you can kind of slice and dice your sets. You can take slightly different approaches at everything. But generally, we're all steered towards this idea of, hey, it's for responsiveness and you're part of a legal team and everything like this. We're going to get to the point where you can say, slice and dice this and before you even have this reviewed, have a subject matter expert summarize this or a cheaper model summarize this.

gigantic document and see if it even matters, right? Think about, you know, we do this right now via kind of some of that statistical culling too, right? You know, you do your filter and define what you're going to send off to it. But. Once we have the entire thing orchestrated to say, here's an entire database, here is the project, come up with a plan, work it all the way through, and then you see it, work it all the way through, you don't need a top tier model for every step. A perfect example came from this week. I have a dashboard for some of the different things I have going on, and it will call back in and show me live every event for any agent or sub-agent as it does anything on a server that I've got.

And that's fine in JSON, but I also want kind of a play-by-play of that to scroll by too in kind of a human-readable way, just so I don't have… I don't have to scour. And that, I don't need Sonnet 454. Um, and so I kicked it down to Haiku, which is faster three times.

You're going to have like footnotes, right? Right.

Yeah. So Sonnet, uh, so let me back up. This has been the case. Yeah. I'll, I'll be a little bit fan boy about this. Um, this has been the case since I started using language models and language and the language model that changed my mind about what the, what their potential was, uh, was Claude versus ChatGPT for a number of different reasons. So I have used Claude as much as I can. personally. And it was difficult to get. You'd get capped out pretty easily. But even early on, I found it to be head and shoulders better. They have different flavors of that. All of these different labs have this kind of thing. And they're frustrating. And if you think the naming and the numbering conventions are bizarre at the consumer level, look at the naming and conventions at the actual model build level.

But they essentially have a good, they have a best, they have a really great, and then they have a cheaper and faster model. At this point, their cheaper and faster model is better than their top tier model was a year ago at a number. of different things, a number of different benchmarks. But it means that if you're sitting there and you are doing a task and you're going to split it up into chunks, you can say, I want one of these. And then you can say, I want one of these. And then you can say, I want one of these. And then you can say, I want one Claude window to handle this task. And I want this one to summarize that work. And the one that does that summary, it can be super cheap and super fast.

But when we're doing it on an agentic level, here's the other thing. I don't even necessarily have to stick with Claude. I can then say, you know, I want to save my money on Claude. So I'm going to kick that over to Gemini. Or if it's just summarizing a paragraph text, I may be able to kick it over to something running locally. You know, it pays zero dollars for it. Zero dollars. So that kind of benchmark that got started out as. I don't necessarily need to use Claude Opus to summarize a deposition transcript, which is what I had in my mind, has ended up kind of playing out at a much smaller scale. But I think we're going to see a scale up again as the agents allow us to scale the power of the AI's reasoning against other tasks.

No, I think that's exactly right. And from when I've been teaching people to try to engage with this and try to get their imaginations to kind of explode to see what the potential of this kind of stuff is, it is easier because you can go to any top-tier model. You don't have to go through some sort of large Byzantine discussion. I can say, look, go check the box on Google and double-check that they're checked on OpenAI and Claude. And anybody is welcome to choose any one, but if I'm going to put my $20 into one, it's always going to be Claude, right? And from there, don't take it and take everything that any marketer or product proposes. Person even tells you that thing can or can't do.

and take your stuff that you've got in your hands and shove it into that thing and see what happens. And don't be afraid. To get your hands wet. Don't be afraid to get your hands wet. Don't be afraid to break it because if you get it and you get a bad summary, what's the worst that happens? You're like, well, that sucked. And then you go back and you read it the old-fashioned way and you finish your email, you know. But you learn to do that. The other thing that I think a lot of people expect is that they need to – I mean, think about this. Typing is a pain, right? Typing is a real hassle. And if you talk to a lot of the guys about how they really effectively use AI and these are these kind of like, you know, these guys that are like, oh, I've used AI to automate my entire life.

One of the first things they say is stop touching the keyboard because you're wasting so much time. Turn on that dictation. I say do that dictation too, but more than that, especially if it's a person that has never used an AI tool as a drafting assistant, not something that you give it a, hey, make this better and rely on somebody else's definition of what better is. To truly sit there and say, okay, chatbot, let's draft this complaint together or let's review this thing together. And when you're sitting there like this, there's so much friction and that is bound up in your fingertips that you don't understand that if you're trying to draft something with. a language model and it doesn't sound like you, you're sitting there and you're trying to say, no, sound like more, you're trying to steer it.

The best way for it to know how you sound is to let it know how you sound in your head. And that's not typing out with your fingers, you know, and that's different. And so I, and that's one of the more fascinating things I think for the legal profession long term is that, you know, especially if you worked at a big firm, you get a new piece of technology. The life cycle is we wait for somebody to say it's okay. Then we wait for security to say no. Then we wait for security to get overruled. Then eventually we get a training package and then somebody gets it and then it's out of date. And what we've been hoping was good ends up really sucking. And we go back to doing everything the old fashioned way.

But this one isn't necessarily that way. And the people that hear about AI and know it's going to change their work, even the older folks, I mean, we've heard this before. They said that with email. And it's like, oh, God, now we've got email. They're talking about efficiency. All my staff is gone. Everything is going to get worse. This is going to be harder. I can barely type my emails every day. And now I've got to learn how to talk to this robot? This is the worst. No. I'll tell you what. If I was in charge of finding somebody in a law firm of Strata, I would walk into a senior partner's office and I would say, you know what? You don't have an assistant.

You don't have an associate. You have this. And I'd plug one of these microphones in there. I'd take his keyboard and I'd throw it out in the hallway. I'd say, bye. But I tell you what, I also guarantee you that within… an hour, hour and a half, he's happier than he's been in years.

Absolutely. Absolutely. It's the old and the new, but also just the old and frankly, just simply the same. And again, it's when we let the product get between the user and the model that you end up having confusion, problems, questions. I've used ChatGPT. You've used ChatGPT. We've both tried to use Copilot. My God. That's the perfect example of the product getting in between. because it's the same brain, but Copilot gets in the way and all of a sudden it turns back into Miss Clippy for real, you know? So any of these models, you've got to get back to the model, talk to it, see what it does. And let's say that whole process of drafting, right? Microsoft says, okay, well, we've got this Copilot, so you're sitting there drafting and it wants you to change paragraph by paragraph.

Paragraph, nah, screw that. Open up that thing and say, hey, I've got to draft a motion today. I'm going to talk you through this and I'm going to read this. And let the thing go. You can let the thing read just garbage text and most people don't understand. I mean, they see dictation and they're like, oh, man, I couldn't follow that. A language model can. Typos don't matter. They do not matter, but what it will do is it will catch your flow, it will catch your words. And within two or three prompts, all of a sudden you realize chat GPT or Claude, instead of talking to you the way that it used to, it's talking to you at your level. And one of the reasons I always liked Claude was I always found that Claude, in its responses to me as I would talk to drafts or these kinds of things, it would meet me at my level of reading or higher.

And it always challenged me. It always gave me lots of good text. And that's one of the reasons that I do read the studies that say people that use LLMs read less, they comprehend less, they learn less. Those people are using it. Certainly differently than I am, I would say, but they're using it wrong because they're missing out on a whole hell of a lot. Because I've read, the more I used language models, the better I got. Think about law school, right? We used to read like machines. Now, we also burned out on it and then stopped. My God, I got done with law school and I went through a phase I read comic books for a long time. That was about as heavy as I let it get, right?

But no more, no more. Once I started reading words, words that mattered, it got your brain cooking and thinking again. And the other thing that I can give a person that is glossed over by so many of the people selling products is that a language model or a series of language models can give a person perspective. Alternate perspective, even if it's different, even if it's hallucinated, even if it is biased. Well, shit, we've biased people all over the place, right? You know? My God. Spotting bias is a critical thinking skill for anybody at any phase of the world right now, AI or not. So we should all get better at that kind of thing. But learning to spot it in writing, in language model, makes it better to spot it in real-time life, too.

Boy, I think that almost any of these tools, once you latch on to one that really works for you, and once you're really relying on a model, and you're building your own systems, or not necessarily building your own systems, I think that's, I don't want to gatekeep like that. And just once you know how the hell you work, or have expectations of any language model, one can be so sticky that you forget to go shopping and realize that there might be new stuff. But at the same time, if you've already got, if you're rocking ChatGPT 3.5, and somehow you're still 10x every one of your competitors, stick with it, baby. You know? But, you know, I have… When I say stick, pick one. Pick one you like.

And stick with it. And again, try to turn all that stuff off. ChatGPT is complicated. I'm already on it. I was still affiliated with more organizations when I started saying ugly things about ChatGPT. It's a fascinating one because they do obfuscate the difference between the product and the model so much inside of ChatGPT.com. And ChatGPT is, and it's one of the reasons it's become so hard, I think, for so many of us as individuals to engineer and get to know. Because ChatGPT was one thing before it introduced its memory feature. And then it introduced its memory feature. And A, it didn't really work very well. And B, it turns out it made a lot of stuff. A lot worse. And the ChatGPT ends up kind of pushing that on everybody.

And they've done themselves a little bit of a disservice because it makes it harder for anybody to learn what the heck vanilla ChatGPT is so that you can take that knowledge of I know how to do this in ChatGPT Pro. If Relativity 1 is running ChatGPT Pro, I should be able to go in and run the same thing and get the same kind of response. I mean, if anybody was opening eyes, I'd say the same thing to them. Open up the system prompt, but I've said the same thing to Relativity 1. Open up the system prompt. Let us see what's happening and let us all work from a little bit more of an equal playing field. Um, and again, that's why I ended up drifting towards the models from Anthropic because they've always been a little bit more open about that kind of thing.

They have much less of a consumer grade. tool that they seem interested in selling. You know, they have, it's, it's, they just have different interests and that's ended up where I've ended up landing, but try to stick with things as, as, as close to the metal as you can. I think that if you are, well, when I started all this, I also want to point out that, that there is at this point, and even when I started this, I didn't, I started embracing AI and researching it and taking it seriously no later than, boy, November, 2023. Something like that. I mean, I mean, things were already rolling. You know, things were already people are already telling me that they were experts. People are already telling me that there were things I needed to sign up for to learn from them about.

And that's just never been my way. But it was also early enough that now you go on Anthropics website. My God, they've got documentation. There was no documentation. It was Claude is a helpful assistant. Bye. And so therefore, almost everything in those very early days in terms of what is this going to be capable of? Right. I mean, all of the things I say I'm doing right now, you know, I don't have a degree in any of this kind of stuff. I got a degree in. You're fucking around and finding out curves. But I did it enough early on that I could a little bit use my intuition to see some of those early hallucinations, let's say. And that was one of the things I've noticed over time.

One of the more – I would sit there with Cloud early on and we'd be doing code, like web code. Nothing fancy. And I'd come back and be like, hey, it doesn't work over here. And I would think about it, think about it, think about it. And I'd come back and I'd say, well, it runs in the virtual machine that I've got. I tested it over here and I tested it using this environment, this environment, and this is the exact test that I ran. And I'm like, there's not a VM behind that thing, is there? And there's no documentation to the degree that it's like, man, I've got to double check. And then I read and I learn, you know, and those were the hallucinations that people are like, oh, we've got to protect everybody from.

So a lot of that kind of stuff that ended up being intuition, but just mistaken intuition that lived in the model itself, ended up getting beaten down. And so a lot of those kinds of early promises that might have given some of us a little bit of clues as to what this stuff is going to be inclined to do later on, have now been muffled out either by kind of guardrails against hallucinations for coding, things like that, or just by virtue of the fact that so many more people use it through a product that you don't get enough exposure to the close to the metal, even compared to what you're used to just going to the website.

Right. It's the critical thinking, but it's the last question you asked me, too. It's trust. It's the fact that trust has to be earned. And we can only give that trust when we apply our critical thinking to these kinds of things.

No, that's exactly right. And that's why I came back. You got my paper. And I didn't learn this kind of country sensibility when it comes to robots from just anywhere. It was here. It's from Colonel Eckhart who taught evidence here, right? It's like, yes, there is every, you could memorize every single rule of evidence that there is and be able to recite it verbatim. Or you could forget every single one of them. And when you are appointed and you need to get into a courtroom and you have to deal with a piece of evidence, you are dealing with three questions. Relevance, reliability, and whether or not it's going to be, whether or not unfair prejudice is going to be involved. That's it. The rules all discuss those three things.

So when we look at. The evidence that's coming out of it, right? I mean, everybody's out there saying, when are we going to change the rules of evidence to fit AI? It's like, why? Is what I'm looking at, as a finder of facts, something I can rely on? Yeah, the question of reliability may be more difficult or less difficult now or different than it was then. But the question itself hasn't changed, right? When we got long-distance phone lines, we didn't change the rules of evidence, right? A lot of other laws changed, but we didn't need to change the rules of evidence. Where we have problems is not with the rules, with the technology. It's in the application of the rules. I hear the discussion about deepfakes.

and the rules of evidence, and I see and I know where it has come up. And where it's come up is theoretically, and then we've seen in security, certainly with the breach out at MGM. And that was pretty cool. I'll explain that. I'll remember. I'll give context this time. All right, so everybody's heard about MGM Grand Casino. You know, somebody broke into that organization and kind of laid waste to it. And the way that they did that was they called into the help desk and they had a reproduced voice of the CEO turning in a help desk ticket to have his password reset or whatever it was. And they said, yes, sir. And there it was. So that voice cloning thing is big and very sexy.

And very theoretical. And we've got a lot of people sitting around at conferences like I've been at, like I have a very privilege of being at. But that's not where this is going to surface. It's going to surface in family courts. It's going to surface in those courts where judges are dealing with 10 family law cases in the afternoon when they've got a criminal docket to run in the morning. Those are the problems that we need to be solving. Those are the education pieces that we need to be hitting at. Not to go sit around and talk about changing the rules of evidence. It's about making sure that we can give our practitioners, whether they're on the bench or whether they are attorneys, truly the tools to do the job.

And to fulfill the obligations of being on the bench and to end up being an attorney. We also tend to stand and we think about the legal process and the legal technology process as a regimented sequence of events because that's how it is. And that's how it's always been to a degree, especially with document discovery, right? It didn't used to be like that. You know, it used to be when document discovery was an afterthought and maybe there would be a smoking gun document. Hell, if there was a document, you know, people would be lucky, right? So it was all testimony. So it was all assumption. So it was all about cross-examination and depositions. Well, now it's all flipped on its lid. But because of that volume, we don't end up actually looking at any of the documentary evidence in our documented process.

Until there's motion practice, there's pleadings, there's everything like this. When we think about that in our corporate litigation context, it's like, yeah, there's no harm to that. When you think about that downstream and you think about how many cases are out there that are decided or end up being resolved or end up not getting resolved merely because that attorney or that case never had the resources to be able to examine the facts and address the true facts. In the legal context. So discovery is about when we finally bring the facts to light. And AI should be about taking that now we're going to work with the facts away from being the afterthought and the punchline, right, at the end of our process to say, hey, now we can get in the thing and now we can go and look at the thing after we've done all this work.

Where this needs to be getting is that the pleadings themselves, the responses, the early negotiations, the resolutions of the disputes, that those are resolved by the underlying facts so that we can get to those underlying facts, extract those underlying facts, and then put them in the hands of the people that can go in and apply them to the law. Because you look at medicine, right? Medicine has a bit of a crisis of staffing and availability. Legal does not have a crisis of that. Legal also doesn't have a crisis in being able to retrieve the law, frankly, right? I mean, using an LLM to go do your research instead of Google and pull your cases. I mean, yeah, it's a little bit better, but if you're just using a search engine, it's almost a distinction without a difference, and it's not solving the problems that we have.

The problems that we have and the problems that clients have and the problems that clients want to resolve is to move this along and spend less of my money and please solve my problem, not let's kind of get into it. So, I mean, think about a world where when we do get to go in and we do get to apply an all-in to your own data, which you know that you can do, that we can do something that reduces the impact of litigation on somebody. So let's say you've got a – one of the things I did at the old job wasn't just railer-related discovery. I did forensics and everything. You both, each of them, have done a little bit of that too. And even if it's in a corporate context and even if it's somebody's company computer, that's their personal space.

Not considered mine. You guys probably consider yours. It's an invasion. It's not somebody anybody likes. And that's kind of boring litigation. It doesn't involve them at all. The thing about the psychological impact on a person if they have something that they want to hide. I think that there's so much. It's easy and it's an easy bale for people to assume that when people have a concern about privacy on their computer that either they're a weir dough or they've got pictures of themselves or their girlfriend or their boyfriend on there. And that's what it is. And that's fine. We laugh that off. And I always did too. But I've. I've walked a path in my life where I have a much greater level of empathy now, you know, for different classes of people.

I understand that privacy means things to different people. And somebody that's in a domestic violence situation, let's say, may or may not be in it. You may not know they're in it, but the fact that they're in it or the fact that just that sucks is something that is extremely critical to them in the act of having to go through and even going through a guided way or knowing that they're in it. Knowing that the company is going to go in, right, and just apply keywords and that might end up in might end up in data. That sucks. This kind of stuff enables something more like this, where a person says, hey, I have got to respond to these goddamn interrogatories and. I'm responding interrogatories to somebody that is abusing me.

Help me do this so I don't have to answer these questions. I don't want to answer them incorrectly, but I do want you to use Google MP, the Gmail MCP. And I want you to go through and help me answer these things. Here are my 40 questions from my lawyer. Please go through my email. Please go through my documents. Help me walk through those. Answer those questions. If you have anything else, let me know where I can fill in the gaps or I need to go find stuff. It goes through. It gathers the stuff. It writes a report of what it did. It goes back to the person. The person hasn't had to talk to any other person, whether they see them or not. They go out.

They find it. If they need help finding it, they're asking an LLM. You know how many times have you guys had to walk through somebody how to back up their iPhone? Right? It ends up breaking a half day if you've got to do it. This person just asked the same thing. Oh, here, I'll walk you through it over here. I'll take care of it for you. Full chain of custody law, full decision law, because it's going to keep track of it all along the way in that person's stuff, in their own personal space. And that person is not going to be traumatized because they're not going to have to necessarily answer questions that are answered by the underlying facts of the documents that are there.

This is where the technology needs to be. This is where people's focus needs to be, the facts, not the accuracy of the law.

Hmm. Right now it's, it's truly finding it, truly try to find the model that will let you be a partner and be able to sit there and make you feel like you're in law school again. Okay. Um, so no, our books here, if you want to go do legal research and you are worried about grounding and the, and the, the BLEX or whatever, being good on the, on pulling everything back. Treat this like we've learned right here in these walls and go find your cases, go out to Cornell, go to whatever, go find your statutes. So the way else we're going to need to remember that these cases are definable, you know, so think about the, the, most of the cases are this kind of way.

Look at the statute, statute may be just enough. We don't need to worry about esoteric kind of stuff. And there may not be a word because it's not some. Highly contested area of law. There's no need to gatekeep a pro se plaintiff from going after their landlord, right? But find a way to make the LLM better at your job in some sort of way that it is competent and you can find it. Because there is leverage there that can help the people that we went to law school who can help. And that's not use this to draft a response to this email. Absolutely not. You want to see the billable hour die? We keep billing for email. Period. Anybody. If it's an email, it would be very well thought and it would be organic.

And if people think AI slop is bad on the Internet, wait till they have emails to have AI slop in their mail because the people at the other firm are just feeding it through. And when you get in a long thread, you want to talk context poisoning. Context poisoning is a sexy new term that's come up, especially in agentic kind of thing. It's like, hey, we're going to put a bad instruction somewhere, a bad document somewhere in this organization, and it's going to throw off their agentic system long term. And that kind of thing is where we're going to see some of that.

Yes.

Well, I mean, back is a relative term. So it's kind of in my backyard. But it's actually more than that. And I did not come to NeLI just out of the fact that it's local. I didn't get involved in NeLI until just a couple years ago. And I have been to a lot of different conferences. And I'm involved in a lot of different kind of communities. And, say, technical communities and other kinds of things nationally. And I always felt that, you know, work was work. You know, you went to some conference. You went to fest. And, you know, same people were all in kind of the same kind of thing. But it really wasn't the same kind of thing that I got at some of the very highly technical conferences that were small.

And there was a… There was a community there, right? You know, it was truly a discussion and a little bit without ego. And when I came to NeLI, I saw that in eDiscovery, frankly, for the first time, because it was that kind of an open forum. There is a conference, there's a security conference that no longer happens called ShmooCon in D.C. And Bruce and Heidi Potter ran it for years and years and years. And, yeah, everybody gave up their, yeah, and gave normal talks, but it was interactive in that if you decided that you were going to talk out of your butt, everybody that showed up there was granted a Nerf ball. It was called ShmooBall, and they were allowed to throw it at you.

And then that person stood up and got to a costume, right? That's, that phosphorous community, that phosphorous, that kind of interactivity. As you get kind of those smaller conferences, one of my very small, very favorite conferences for my security is kind of… At the time, it was kind of invite only to come, but it was actually held as like the best Western. And it was amazing. But it was because there was very little ego. There was very little on display. And I don't know whether it is, I think it is an assessment to the organizers and a testament to this town and the people that have come and the people that have still come. That this feels different. It feels like I'm going to come and learn.

I'm going to have to have a discussion with very, very smart people who are there to have a smart discussion and not tell everybody who they work for.

Thank you, guys.