Improving Document Review

Thanks for having me. It's great to be here. Now before we get going, Rachi, it's worth actually saying that you not only came to the podcast recording studio here for us today, but You are located in Israel, and you literally just got off of the plane. From America, it's ten o'clock at night you are jet lagged and yet you are talking to Jared and I.

Yeah. Absolutely. Happy to. I mean, chance to talk about myself. Sure.

I got started in legal technology almost twenty five years ago, like most people by accident. I am not an attorney, so I had no true legal training or legal background. My dad was an attorney. My Aunts and attorneys, my uncles and attorney, and they all told me that I was not to go to law school.

They gave me direct instructions not to get involved in the law. So I avoided it, went the technology route, and ended up in Wiggle anyway. But I got started working on a project for Meryl Corporation, working on a government project with the small business administration where we were scanning in loans that had been made over the course of forty fifty years and scanning them into a big database, where then underwriters came to underwrite those loans, And then we went ahead and have those loans available online in order for the investors to come and look over those loans make comments and notes about those loans and then eventually bid billions of dollars to buy them from the SBA.

In order to service them. Well, that process sounds very familiar to anyone who is involved in early days of discovery and scanning and moving everything digital. So when the small business administration eventually got into a fight between the office of management and budgeting and the general accounting office over the profit margin on that project They both agreed it was highly profitable, but they had a difference in opinion on the origin. They shut the whole project down until they could figure it out.

So, Merrill, my boss at Merrill said, Hey, we've got this legal division that does very similar things to what you're doing. And they could really use some help. Would you come join over here? And I said, I don't really wanna go looking for another job.

So sure. Let's go for it. And that was my entry into legal technology. From there, I worked at Merrill for a couple years.

Eventually left. I joined the founding team at DiscoverReady in two thousand four when DiscoverReady was getting going. State of DiscoverReady until two thousand eleven twelve, something like that, when then I moved over to join the management team at Equivio. For anyone who doesn't know is a text analytics on the compliance and legal discovery space doing everything from the originated email threading and near duplicate analysis to predictive coding and all kinds of analytics tools.

I went ahead and then in two thousand fourteen, we sold Equivio to Microsoft where I then joined the Microsoft Office three sixty five at the time. It was Office three sixty five, not M three sixty five. But the Office three sixty five compliance team helping to design and manage their eDiscovery and compliance tools for corporate clients within Office three sixty five. I spent almost five years on the product team and then for my last two two and a half years within Microsoft.

I switched over to helping to manage the eDiscovery and legal technology for Microsoft legal. So going to the corporate side and helping to manage discovery there. Did that for, as I said, two and a half years, and then about a year and a half ago left to start a new venture, startup called Altorney, along with my brother Shimmy, We have we designed a marketplace platform for doc reviewers. Basically a gig economy approach to doc review.

And that's what we've been doing for the last year and a half. Thanks for that, Rachi. I mean, you've been doing this for a long time. I think we can consider an OG. Oh, man. You're gonna make me feel old now. Come on.

Oh, what excites me? There is always something new, always something a new problem to solve, something that people haven't thought about and aren't gonna think about. You know, legal technologists by nature.

They're adaptive. They need to be able to go into any situation and free think and just figure out how to either apply existing technologies to a problem or to come up with something totally new to solve an issue and that's what keeps me going. It's that ability to always innovate and always look at both the outside influences of what's new, but also the internal influences of what's new and coming up with solutions there.

Absolutely. Because when my brother and I had decided that we wanted to do something to get We had worked together when I was at Merrill. I actually hired him to join Merrill when he was still in college and We had always decided that we wanted to do something again together. And when I knew that I was ready to leave Microsoft, We sat down and basically put up a whiteboard of about twenty five different ideas of what we could do.

Everything from, hey, we could start a new service provider. A advisory company, design new analytics tools, you know, come up with a new review platform, and then about ten different ideas outside the industry. But what we really wanted to do was based off of my experience at Microsoft and really a combination of Microsoft and going back all the way to the DiscoverReady days. They wanted to do something that could help a large percentage of the industry that was innovative.

It was something that people had it seen before something that was new and really could have a impact in a positive way on not just the corporate clients, the people who are spending the money, but also on some of the under service members of our community of the legal technology community in general. We're going pretty broad here in terms of how we defined it. So We knew we wanted to do something like that, something that would keep us interested in something new. I had seen, you know, firsthand bringing on doc reviewers through the process that there wasn't much care given to who they actually were and what they were doing And overall, the drive to lower price push their rates down as far as you could possibly imagine.

So we saw something that we could potentially do by using technology, to get rid of a lot of the overhead that exists within the process there and both save clients money, but more importantly, to give more back to the reviewers and create a better environment for the reviewers. We weren't gonna touch anything that had to do with data. Weren't gonna touch anything that had to do with the work product or the content that the reviewers had to do. If we could help it, but we felt that we could still use technology in a way that would better the overall community there and have an impact on quality.

And you feel that we've come up with some really interesting ways and ideas to do just that. It's been a lot of fun. You expand a little bit about how you're tapping into some of those underserved communities? I find that fascinating.

I mean, the first thing is you gotta look at pay. Right? In recent times, reviewers, there are reviews that are going on where they are offering reviewers in major cities twenty two to twenty five dollars an hour. And, you know, if I'm an attorney, I'm not, but if I were an attorney who's spent all the time and the money to go to law school, and this is my passion.

I want to be able to practice law, getting paid at those kinds of rapes, I should be working at a big box store or something like that. It is crazy. So the first thing to attack is rates. Now, we're never gonna be able to take a twenty five to thirty dollar reviewer and get them to be paid a hundred and fifty bucks an hour doing doc review.

That's the key. But we can absolutely, by, again, leveraging technology to reduce the overhead, we can raise their rates by a couple of dollars. And, you know, a couple of dollars here can be a decent percentage of their take home. So that's number one is look at rates.

Number two, is look at some of the benefits and services that they get offered. So that even though they are a gig economy type worker, you know, most of these top reviewers, even if they have at work through one company, but they're bouncing around a bit, and they never know even if, you know, they're working for a single company, They never know that their next project is gonna actually be there and when it's gonna start and when the coin's gonna all of a sudden pull the plug on an existing project and all of those So how do we give them more transparency?

How do we give them more insight into what's actually going on? So we've done a lot of things in terms of what we know, meaning whatever the client is telling us, right, or actually putting into the system because even though sometimes I'm a people person, but we don't wanna talk to people. We wanna let the technology do its job. And so whatever the client is actually entering into the system, We let the reviewers know that.

So for example, when they're adding more reviewers, when they're reducing the number of reviewers, when They say, Hey, I need ten people on this project. I'm gonna let them know when they're changing the dates and they're adjusting the number of documents enough to review and all of those types of things. We're gonna let the reviewers know as much as we possibly can about the projects to give them as much transparency and as much insight into their lives. Again, a little bit more control over what they're doing.

But then the last piece is How can we upscale them? How can we give them more tools to do things that will actually get them higher rates and give them more rewarding work There are plenty of doc reviewers out there who choose to do doc review. Absolutely. They like the flexibility.

They like the diversity of the content and the topics and absolutely more power to them. But there are plenty of people who don't want to be doing doc review full time. They want to try to figure out how to get out of doc review. So there we can look at things like mentorships.

We can look at things like the same clients who are looking for someone, the corporate client is saying, hey, I need you to reduce the bills, and we're not gonna pay associates to do doc review? Well, those same people are also interested in reducing rates when it comes to things like Depoprep, when it comes to large scale, you know, custodian interviews, when it comes to other types of legal work that I mean, I hate to say it, but it's grunt work. You don't need to be actively involved in the strategy of a case in order to perform a study and interview and get that information. Obviously, it depends on who the custodian is.

But for the vast majority of it, I can have a lower priced resource than my associates in the firm or even my paralegals in the firm doing this type of work But to a doc reviewer, that might be double or triple what they would get paid for a standard doc review. So how can we do things to create paths for the reviewers to be able to upscale and do more rewarding work both from a professional perspective and from a pay perspective.

So all of those things are just some of the things that we look at in terms of how we approach the reviewer community and how we're trying to position the doc reviewers who are on the platform to be able to basically improve their lives.

It came down to a couple of things. So first of all, it was just looking at everyone who's involved in the process no matter how good technology gets, the need for document reviewers for humans to look at data remains. And they can be involved in various different levels in terms of if they're training AI models, if they're actually doing linear doc review, whatever it is. But that those people are just considered quite frankly the lowest tier within the eDiscovery process is ridiculous.

They're smart people. They're intelligent people. They've, you know, gone through the training. They know what they do.

Some of them are jaded at this point. Unfortunately, But just the fact that they are looked at as a commodity, right? We don't look at eDiscovery project managers as a commodity. Absolutely not.

Right? We don't look at, forensic tech as a commodity.

Right?

Why are document reviewers looked at in that way. So that bothered me a lot. But there was one other piece that I felt when I was looking at the process and looking at what's happening here. It is accountability.

There has been in general terms. And obviously there are companies that do this differently and handle things differently, but in general terms, There is no accountability on either side when it comes to doc review. There's no accountability by the reviewers because If I'm a bad doctor viewer, I don't really care. I'm gonna get hired on this job.

You know, they're not really looking, and I'll make it through for a couple months, or I'll get fired after two weeks. Right? Because they finally got around to looking at what I'm doing and, okay, they want me off the project. Well, a staffing agency is just gonna put a moment on the project.

And maybe they get kicked off that one. Well, okay. There's thirty other staffing agencies that I can go to. And I'll bounce around and continue to get work and pretty consistently until, okay, I come back to the original one because Half of the staff there that remembers me is gone.

So I can now get work there again. There's no accountability from an industry perspective there. But on the reverse side, there was no real true accountability on the client side. And I don't care whether the client was a law firm or a managed service provider who's offering manager review.

There was a general feeling of reputation. Right? You know, that's a great firm to work for. That's a horrible firm to work for.

A great company, whatever it is. But other than that general feeling, if you have a company or a firm that, you know, the managing attorney who's managing the doc reviewers, is yelling and screaming at them all day because they're just a commodity. I don't really care about them. Right?

And if I need to, I'll just replace them. Well, Why should anyone have to go through that? And by building out a proper two sided rating system, and a lot of research goes into how to actually do that in a successful way that actually can improve a market, but we felt that we could actually use that as a piece of it to improve the quality that a reviewer experiences.

That a reviewers going improve their overall work, right, give them incentive to actually improve, but give also incentive or usually disincentivize the companies, whether they be, again, managed providers or law firms to avoid and train their people on how to manage doctor viewers because Without that, the industry, it just becomes more and more of a commodity and the situation gets worse and worse. So we really saw that this use of technology could truly impact how the quality and the life of reviewers and the quality of projects because that's the only way that we're gonna improve is if we work together as an industry, and that's why Jimmy and I really felt that we could come to this, not as a service provider, right?

Because if we were a service provider doing something like this, well, no one else is going to use it except for us. But by offering a industry wide platform that actually gathers intelligence for the industry and offers things around knowledge and think about it this way. If I have a reviewer who is ranked four point seven out of five. Right?

And we know their metrics, and we know things about them that show that they are actually a great reviewer.

Right? Versus a reviewer has rated three point five and is an average rate. And twenty five percent of their work to be redone after they go through it. Well, I can then go and justify paying the four point seven reviewer eight bucks more than what they would even normally get. So if they're normally getting thirty bucks, thirty two bucks, I'm gonna pay them forty because It's worthwhile from a project standpoint and the client can see that difference. There was nothing out there that told me that that reviewer actually deserved it. So if we could build that, that's where we saw opportunity to actually impact this community in a really, really positive way.

So it's a combination of all of those things that add to the accountability that end up again going back into the reviewers pocket benefiting everyone, benefiting the reviewer, benefiting the client helps to make sure that the industry as a whole is improving as opposed to just posted.

It's a great question, and it's one that we get asked pretty often. And I discussed, you know, as someone who was at Olivia, who helped to build Equivio and into what it was. I'm a big proponent of machine learning and AI from, you know, the use of technology when comes to document review and litigation technology overall. So why in the world would I get involved in something that seems inherently against that.

It's the people side. So all of the things that I mentioned earlier play into it, but the true is that we, as an industry, have been talking about the use of, and I don't care if you want to call it AI machine learning, but how do we actually apply technology to document review in better ways? Right? We've been talking about that for years.

The use of predictive coding since two thousand eight, two thousand nine, and the use of now, you know, trying to figure out how to actually use generative AI in document review, which I think there's a place, and we can talk about that for a couple of minutes, but we've been talking about that for a long time. And there's gonna continue to be improvements and new technologies that come out that will help us to improve how we do document review. However, for all of that discussion about how predictive coding and AI is going to, you know, totally eliminate the need for document reviewers and totally eliminate the need for attorneys to ever look at data.

We have more document reviewers in the industry looking at data than ever before. Alright? We have more cases. We have more data.

Now Do we still have those, hey, I need three fifty attorneys sitting in a room or nowadays remote on a single case? That we used to have those, you know, huge massive matters that anyone who has been in this industry has worked on. Some of those, if they have been in the industry for a while, No. We don't have those massive matters anymore, but that massive matter that had those three fifty attorneys on it now By the use of technology, we get that down to twenty reviewers, or fifteen reviewers, or ten reviewers on that same exact manner.

But whereas before there was one or two of those a year or whatever you wanna say the actual numbers were, now there's thirty to fifty of those, but only with ten to twenty eight attorneys. And I think the use of generative AI, if we can figure out how to actually make it used properly to help to call down the data and understand the data.

Is going to help us to again reduce the number of people who are needed on any given case. But the number cases and the amount of data is going to continue to increase. We all know the statistics. We all see the, you know, hockey stick growth of data across corporate world and within discovery specifically.

It's just gonna continue to increase. So we have to use technology.

We have to look at how can we use these types of technologies to help to reduce and better understand the data, but that is not going to totally remove the need for human involvement. There's this new concept that's come out over the last, you know, year of a prompt engineer It's someone who truly understands how to put in the right context into the AI chatbot in order to get back the proper results.

Right? Well, that's an opportunity that document reviewers are going to have. There are people who understand what they're looking for within data And quite frankly, the partners at firms, they might be able to come up with the general questions of what they want answered, but they're not the ones who are actually gonna sit there. And start really truly typing in or speaking to or however they're gonna communicate with the AI They're not the ones who are going to spend the time to truly understand how to manipulate and get to the results that I actually need.

Now are there gonna be exceptions? Are you gonna have, oh, you know, that partner, that associate, whoever it is, you know, or at an AOSP? You know, oh, people at BDO, they are awesome at doing that. Sure.

You're gonna have those people within the industry, but I see it as something an opportunity that document reviewers can actually get into to truly help to bring this industry forward with the use of the technology. So that's really interesting. So what I'm hearing you say if I'm understanding, right, because it's fascinating. What you're saying is that as the data continues to increase and as technology continues to increase, And by the way, as a footnote, I will make the comment here that I think that the generative AI, I think that this is permanently here to stay.

To review, even with all the tech. Is that what I'm hearing? That is absolutely what I'm saying. And so even with generative AI, and we're gonna find out what it's gonna do in either eDiscovery industry or in the doc review industry.

They have the skills They have the intelligence to be able to do it. We just need to be able to train them and point them in the right direction to help them get there. Now the truth is for as much as everyone within the entire industry right now is talking about, oh, we've implemented generator AI entire platform, blah blah blah. No one's truly come out with.

Here's how it's going to really work within a eDiscovery platform. Like there's just no one out there at least that I've seen that's truly implemented using the new generative AI tools within a eDiscovery platform form in a way that truly works at scale. So we still have a while to go. Innovations happening faster than it's ever happened before.

So it's not like oh it's gonna take eight years until we're at the point where people can start using it. Over the next you know year to two years, we're going to see it truly implemented within platforms and not necessarily within the platforms that we use today could be within totally new platforms that are focused solely on this. We'll see what happens there. I think that it's an opportunity for an not just doc reviewers, but I think doc reviewers are primed to be able to do this if they want.

But we have to give them those opportunities to be able to build it into something that they're ready when things get down to. Well, I really only need one attorney on this or two attorneys on this I don't even need that team of five to ten. And we will get there where it will only be one to two attorneys on a case and where you used to have five to ten where you used to have fifty to a hundred where fifteen years ago, it was three hundred and fifty. We'll continue to see it go down, but we're not gonna eliminate it at least in my mind.

First of all, one of the things that I found fascinating is I've got what I think is a pretty decent set of connections within the industry. It is nothing compared to what Tom has. And the people who were able to pull on through his connections has been awesome and to hear all of the different journeys because we always start off kind of similar to how you started off here. To hear all the different journeys how people got involved.

And we're talking to everyone from judges to partners and firms to people who, you know, are on the technical side just across the board. How everyone who's involved in legal technology and discovery, no one set out to do it. As we go, there's Robucky that we have a couple law schools today that have some sort of eDiscovery symposium or something in their curriculum. So that's been fascinating to just hear all the different paths that have come to play here.

The other thing I think is There's a common goal that you see through everyone that we talk to, which is the interest in moving the industry forward. No one's sitting on their laurels and it gets back to my original comment of there's always something new to solve. There's always something that's challenging that hasn't been seen before and listening to how a lot of the different guests about and their approach to these challenges this fascinating. I actually recorded a video cast for this afternoon.