Building Better AI Programs Together

Absolutely glad to be here, Daniel and Eric. Appreciate joining the friends here on the cast.

That's a great question.

I definitely am going to first state it was not solely my responsibility. I was simply acting in sergeant in arms, if you will, to some very fine executive leadership. I basically saw some things unfolding in the market and the industry, and let's start investigating and understanding this. As opposed to following the hype cycle, let's really kind of get in and unpack it.

So I'm an operations team person, right? I run a team of people that are working with the data and moving things through or have done things all the way from paper to, you know, the electronic version of the world of how do we gather data and information, make sure it's secure and it's compliant, and then ultimately feed it in for any type of review and production. So it comes up to me and the team, quite frankly, to say, alright. We know what we do at the core, right, as a legacy core, an accurate level of offering of process and documentation.

How does this AI tool, this fancy thing, how does it make our experience different, better, not so better? You know, let's read through the hype a bit. Without that executive leadership, specifically Joe Shanker, he definitely captained the charge, and many foreign partners that were in that league also.

We have the marching orders, and it's always easier when you have the marching orders as opposed to trying to wag the dog. Right?

Well, basically, when we were looking at the process that we already had to follow and we said this new technology is intended to be the substitute or the next version of TAR, that was kind of the natural place for it to fit. So it has models, it has things that can create classification, there numerics to it. It's similar to what we already do with TAR 1.0 and 2.0. Right?

And so what we've quickly found was that the ability to classify the documents very quickly and accurately was happening in a much shorter life cycle. So less times working through a bunch of different versions of control sets, all sample sets, all that stuff. And yet we still had a pretty strong validation process through it. And it didn't happen overnight.

There was definitely some hours of like, well, that didn't quite go the way we thought it was gonna go, so let's make sure that we're, you know, applying the proper validation downstream and or tweak what the features are in the application and or the process to make sure we're getting what we're looking to get and find and identify the efficiencies. So a lot of it's just making sure we take advantage of all the rich experience and nuance that we're used to dealing with in these projects and apply that to this new technology and then really have a couple of strong developers who are committed to the cause.

You know, we spent some late nights just trying to get through the process, layer in workflows to do things that will help make the product more efficient.

And ultimately, with that executive sponsorship and even a couple of clients that were on board, we were able to get some real strong use cases in pretty early and really start demonstrating, oh, there's something here. And so once we were there at that place where the seed was planted, we then organically grew.

Still trying to follow and make sure, you know, we're doing things in a way that protects the client and the data and the process so it's defensible.

Yes. And it's some people, it didn't foster immediately. Right? You always have people in the room that are pretty comfortable on the edge and saying, well, I'm gonna try these things and see how it works.

And then others that are a little more comfortable in their safety net of I know my approach and I can be confident that it's gonna kind of get the same results I've been doing for the past ten, fifteen years. So there's definitely different tribes and people that but we brought people along. I think what we tried to do is just create a culture of and this is very much helped by the firm that said, we are gonna create safe spaces where we're gonna use AI. Now, you can personally use AI however you want, right?

You can go out and create travel plans and create your own diet program. So I also encourage that. We encourage that with the team. It's just like, just understand what it is that these tools do and get comfortable with the form of prompting and asking the question.

And then if you start using different tools, understand if there's variables or differences. And this became really important as the models themselves are competing aggressively with each other and things like multimodal came out and the ability on translation and all this stuff that was super helpful.

I went on this big trip to Japan and the first thing I did was ask for, like, perplexity, create a whole trip, da da da da, that my two sons are going with me, so throw in some anime and, you know, within seconds, I had this whole layout with maps and the train schedule. So that level of kind of, like, accessibility is kind of mind blowing. And once you start opening and triggering that creative side of it, you realize there's some possibilities here that are just applied in your day to day work.

And it's creating safe spaces with those possibilities, safe meaning, obviously, safe for client data, safe from security standpoint, then people start getting creative on their own.

One of the things we've done, we have Enterprise ChatGPT as a firm. So it's pretty much standard business now for us to throw together a project or a GPT to help kind of navigate through specific processes within the department and or document repositories that we have. So we can just ask a question, get back the feedback on our standards or our process, and we're off to the races, which would normally be, you know, go look at this place, go look at that place, go look over here, and then bring it all together into, you know, some type of summarized document. But now that's all in one place. So that's just that's one use case that's just become very routine. And once you get people in that space of, well, what else could I could do? Then that's where you can see some real wonderful grass is greener situations that you can at least test out and see if it becomes more sticky with other people in the firm.

That's interesting.

What we did was and you're right. We do still have a standard of legal project management in which we operate of we're just not gonna let processes just kind of sit in silos or this little bubble, you know, runs their special little process. We we try to have a contiguous process across. And we already have a project tracking and a pretty rigid project management platform that involves documentation and connected to tables.

And we have a actually subscribed to a SaaS solution that is a full project tracking solution. So we're looking at ways that we can either AI will plug in as being a resource to retrieve information and generate information across that data source. And then we're looking at ways within talking to the provider about can we enable AI to be more agentic within the platform itself so that it's helping just create more efficiencies in the workflow, but still human in the loop as a part of it. And when we're looking at these kind of use cases that are more back office, there's a framework for which we deploy any of our technology, whether it's integration or developing new tech, developing some type of migration process between one solution and the next.

We basically have a short form program that says, what are the requirements? Let's understand who all the stakeholders are, typical LPM kind of stuff to basically say, alright, we're doing this thing. We keep a repository of we know that we're doing this thing and it accomplishes these types of tasks. Now let's resource that across whatever other projects we're working on to make sure that we're not creating any redundancies or unnecessary duplication and make sure in real time we're looking at efficiencies.

So if in this project that's dealing with how we're generating budget summaries, we could be repurposing that to how we're maintaining our preservation and lit hold process.

We're just completely transparent about that across kind of a single source of our dev program.

So that's the framework as it sits now so that we're keeping everybody educated and aware of the new stuff people are trying and where things don't work, more importantly. And then so when we're creating the next, okay, we think we can improve on this, we're doing it from a place of transparency and education from real experiences and not just kind of conjecture and we think it can do this kind of thing.

One thing we don't do very well in this industry, maybe as human beings, is we don't share very well. We try to differentiate ourselves by bringing forward this super cool thing that I built. Well, okay. Well, teach me how to do it. We'll be a better stronger team.

Well, little Daniel.

Essentially, multimodal, in my layman's terms of it, is the ability to process multiple languages as well as the ability to handle multimedia. So you're able to handle images, videos, and be able to gather and or capture information related to those types of objects as opposed to the early stage LLMs were focused just on text.

Agentic, this is an interesting phrase that's really we love a good hype, and I think the entire investment industry loves a good hype too.

And what we've come to understand is it's basically looking at and each platform kind of defines it differently in structure and how they kind of deploy it. So Microsoft three six five or Copilot might have a different approach in how they tackle this versus what OpenAI does.

But in essence, it's basically creating many program tracks that ultimately may work or interact with each other. And the concept being that it would automate tasks and or workflows for you and or possibly make decisions. But this is where it gets really gray about how much do you let it loose and what kind of governance do you have on top of that. So I think that's a big part of the conversation that's happening right now, like letting it do something so it handles itself as an agent and it's doing decisions that normally a human would do, there's some fear without any type of oversight.

And the thing that I think most people are worried about is if, one, the people that are not as governed and don't really care about impact downstream are gonna create some really cool authentic tools that do all kinds of really cool stuff, but your data is probably getting leaked and or there there's are some compromises that are probably happening downstream because they're just not worried about that.

We, on the other hand, are hyper worried about that. So much so that I don't know if you're familiar that some of these standards that are starting to come out, so like ISO forty two thousand one, I just got an honorary certificate, so I've I've passed the first level on that. And we're pursuing that full certification in line with our other ISO offerings. So I think the NIST has a framework also. But this is a space that's starting to really evolve, especially considering how fast the environments are moving and developing themselves. So it's so important that you have a framework in which you deploy it, you test it, you continue to put controls around it, you make sure there's a strong security architecture that all kind of make sure that the safe places that you built today continue to be safe places.

Going across multiple sources and dealing with how you know the process of getting to those sources in the first place is still, this argument task. In the end, all parties wanna know everything as soon as possible that's wrong or, you know, where the high risk data is or information is so they can start making decisions. Right? I think everybody in the room wants to have that information. Getting to the information as you kind of just described is a matter of effort and time and what do you have access to.

Ultimately, if you can create an environment where they can get access to that information that much faster, I think this is where a lot of tools are trying to get as far left as they possibly can, you know, in the EDRM where the information lives in state to be able to get to that information so they can make the business decisions they need to make.

But we all know that the lit support process is about making sure that you do it in a way that's defensible, it can be used as actual evidence, and the evidence itself is accurate. So those are all things that have to be controls in place to make sure that they can make the decision on the thing that you're ultimately telling them is fact.

I insist, just even for myself, to make sure that I have an understanding of what it is that is happening and how the tool is providing value. And I wanna see how things are kind of how it's making the sausage a little bit. Right? And so maybe some point in the future, you know, just like air, we're just so reliant on it just kinda works the way it works every day, you know?

But as of right now, I still need to better understand how is the framework of the value that you're outputting to me, how did it generate that, and try to do that in such a way that I'm not fully exposing the vendor or the solution provider's IP, but in the end that that I have a I can explain when it misses it for whatever reason.

I think that's what we spent the lifetime trying, you know, make sure we build solid processes around so that when we're outputting our final set of productions or final set of data, we can stand behind it and say, this is everything. This is everything that we've said we were gonna do and got processes and logs to support it. I think one of the things that we all need to be aware of right now is when you're going the prompt approach, you're not getting all the data. You're getting a summary of what it considers to be the most relevant data. And so there are use cases where that's important.

But in the end, if you need to do a fulsome report, you need to make sure you're architecting your solutions such that they output all the variables that are in the data source that you're extracting from. So that's where you're gonna still use traditional tools that you already have in place, including old school machine learning techniques. Just some of the things that, you know, I try to look for and always keep that human in the loop.

Yeah. I mean, that's definitely a burgeoning. Like, solutions are coming to the market to do some level of forensic analysis. And then there's forensics professionals out there that have some tools to be able to do that, but that's an emerging market, quite frankly.

There was one tool that I looked outside the spectrum a bit of just let's porter in the legal tech space, And if you look at, like, data privacy slash protection services types of platforms that do basically individual brand protection, things of that nature, they have some tech that's really kind of next level that's there's one solution, I think, it's called Clarity that I was looking at. That's all the space they're in. They secure high wealth individuals and their content, and their ability to say, okay, that person can authenticate any commissioning of, you know, checks they're asking to be submitted or values that they're asking to be submitted through this authenticated type of process.

So somebody can't just spoof them, simulate that, which that you know, we heard about that case, I think, last year where somebody, you know, passed off a check or something or commissioned some type of massive distribution through a spoofing attempt. So I think that's something that's coming along. We haven't had a high volume of that type of data, to be honest with you, that had bring it to question. I think ultimately the process would probably be is whittle it down to those five or six that and we need to better understand and authenticate that they're actually real, and then send them to an expert to at least do some level of analysis to authenticate their actual forensically sound.

Well, we do have a policy, and there's a framework for which you are able to use AI and sanctioned, basically, AI products that permissible.

Using other ones, you're using it at your own risk if you somehow take the data out. But, you know, those are things that are just completely discouraged. You have to use things within the framework of our security environment. So that's just discouraged altogether. I think what we're trying to do is promote people to come up with the ideas and figure out ways to try those ideas within the safety.

Yeah.

Think it's continuing similar to what you do with data privacy and some of the other types of frameworks you're trying to help people protect themselves from themselves. You say, here's the places where you can play, and this is how you can play with them.

We do have rules that are given to us by our clients about the use of AI, so we make that very transparent and we say, there's some no no's and there's some ways that you can use it, but ultimately, let's talk about what you wanna do with your ideas. I think it goes back to our earlier conversation too about silos that can exist. And if somebody's creating a lot of really cool or learning at least, the very least learning about what doesn't work, that should be shared. So we try to create workspaces. I know our team, the EDLS team, is one primary workspace where we're just looking at ways that we transparently are utilizing the tools to make life a little easier. I think one thing that's been interesting, and I'll put this as part of the conversation point, is we're getting a lot of direct questions from clients saying, how are you using AI?

So let's talk about using AI. And now it's very proactive of we need to know that you have a program in place where you're at least fostering a culture of trying some things out in AI. Now, the thing that's next in line in that question in the line of reasoning is, can you quantify the value? It was really easy to do that with, like, TAR because we have, you know, tens of years of data and thousands of projects where we measured TAR.

And we measured, you know, how fast we got through projects doing just manual review, let alone all the iterations of TAR. So we kinda came to the table some rich data there. But you try to do that with, like, DTEs and value propositions within the legal work, the lawyers' advisory services, that becomes a different quantification. So those are things that we're we're chipping at.

I'm curious, are you guys seeing the capability to quantify that? Are you just kind of starting organically? How's that playing out for you?

It's good response. Yeah.

There's that as well. It sounds funny when you're talking about an AI, but it's true. So I think that I ask better questions. I think Eric asks better questions of humans as a result of us having to be forced into a mode in which we are substantially more thorough in our prompting.

Substantially more thorough.

That's very interesting.

Correct. And I think they know how to just from the Socratic method and how they've been taught in law schools, that is a method that do come out. I think, kind of thinking in that framework, I agree with you a hundred percent that it has improved my communication styles and the precision in which I'm asking these questions so that I know that I'm, you know, more accurately framing what I'm asking for. I think, and I completely agree with you that this kind of iterative train of thought that you kind of start with this initial prompt that's kind of framing things out with some context, right, of this is what you're looking for.

And then I think the best thing that the models have really starting to master is their follow-up suggestions and questions.

That's life to me. That's the deal breaker because ultimately, if it continues to accrue, I know Stephen continues to kind of look at this from this perspective, so I'm gonna keep on feeding him because I know where he's going.

That's all our short learning curve with our communication style with any other team member, right? It's you go to that third year associate you've always worked with, you've done projects with them for on this particular subject matter, you don't need to teach them about everything else. You can literally say, this is what we did with the last project, remember? It's kinda like this, but I want you to give me more of that same, but throw on these three other variables.

They come back and they say, alright, cool. This is what you did before. This is how the things work. And you take that collective knowledge that the models continue to kind of bring into and retrain on, that just empowers you that much more to be able to drawing in your ability to navigate through information, be able to tell that story, be able to tell a story for different types of people.

I really love framing it to I did one where I said, you know, put me as a McKinsey analyst and do this full breakdown. I asked for this particular use case, looking at a pretty strong kind of comprehensive market analysis. Then I said, break it down like I was a twelfth grader in school. And then it just really super simplified it.

I kind of went in between there and iterated, and my ability to output those things, it just makes it that much faster. And I know you've both had these experiences with how you've kind of helped with your consulting and managing your day to day, either internal or external projects.

So in referring to the account, I think you're what you're talking about is of just building out a model that basically kind of records the things that I version of the And it is absolutely, I would have a disclaimer on it, and it'd be more for my great great great great great great grandchildren. And so, you know, because I've done the opposite. Right? I've gone upstream, and I cap out at a certain point. Right? I think I can go back two hundred years.

But theoretically, since the information age, right, we have the capacity in you know, we're not burning down libraries anymore and entire genealogies disappearing like in parts of other foreign countries. We tend to retain information. If anything, that's the toxic waste of our time.

Your question is right of which is the accurate information or the accurate version of the information. I'm trying to make sure it's a private place where I'm capturing it so that ultimately I can somehow IP it or lock it in and say, this is what Stephen was thinking and how he kind of viewed the world and or states the facts about himself, right? So if somebody said something different that, you know, I didn't go to college or I didn't, then it's like, no, the fact support it because I have, you know, I have the college degree, the college has my degree.

So those are the things that, you know, ultimately my goal with it was to say the great grandchildren could have some point of reference because I'll never have that point of reference for my great great grandfather.

And what will that enable for the human race, honestly, to learn from experiences? Because I'd hope I'm a big fan of just share the information.

That's the important part of our human evolution. So if whatever I can do to contribute to that, say, I think there's something here to be learned, that's my little piece to give back, I guess. You know, hopefully, it's people benefit. If people wanna control information and say, I wanna spin it towards a particular agenda that supports something that I believe in.

Absolutely. Really appreciate you guys. I know both of you. So you're solid people and ultimately, I'm glad to be a friend to the cast. You know, just keep the conversation going and we're all learning together.