Dr. Jeremy Pickens on Search, Relevance and the Road from TAR to GenAI

Thank you for having me, Daniel. And if I may counter and say I'm… I'm just happy to be part of the flow of eDiscovery. Any ideas or any thoughts that I have pushed forward have their precedents, and they have their antecedents as well. I had a colleague that I worked with many years ago at Fuji Xerox Palo Alto Research Lab out in Silicon Valley who used to say, your ideas, whether or not they are yours, will be stolen. This idea that we all borrow from each other, we all contribute to each other, and so if I have had some part in that, I'm just happy to be in the flow.

That's, that's a good question. The perspective that I looked at it from when I was doing the work back then was, was less so on the thumbs up, thumbs down, hear songs you like and don't like. And it was more on the sense of, are there structural patterns within the song, the actual, Harmonic progressions, the notes, the rhythms that can be traced back to earlier works, right? You can see sort of rhythmic similarities, harmonic similarities in different pieces of music and thus see how a composer may have been influenced or at least be somewhat related to work that has gone on in the past. And finding those connections historically over time between various pieces of music, I think, is very similar to the kinds of storytelling and pattern finding that we want to do in eDiscovery collections.

You know, what is the story? How did it evolve? How did it change and develop? And can you find historical precedent for something that was said now with something that was said half a year ago, two years ago? So that, that's where that intersects. Uh, you know, of course the idea. of thumbs up, thumbs down is more, is a more general machine learning, supervised machine learning approach. Uh, but I liked the, the music perspective, uh, gave me the ability to think more about the evolutionary patterns, uh, not so much just the, I like it or don't like it kind of.

Oh, goodness. Wow. Great insight. this it something I haven't dwon yet, so I don't want to talk too much about it, lest I give those same competitoors who made many more millions than I ever did some ideas. But, you know, think about the short messages, you know, Slack teams, text messages. You know, won of the big problems in music was where it the boundary, where's the semantic boundary of a word, right? An individual note by itself means nothing, right? An entire piece it too much. But it their a concept of a musical phrase? Yes. Sure. A short motif, you know, a set of notes together, and where exactly it that boundary that gives you something semantically interest enough to want to look for that same pattern somewhere else?

It's the segmentation problem, right? And it's, that's the same problem we have in short messages where you have this stream, not only do we talk differently, but you have these, they're short messages in little bursty patterns. And when we try to break them up, we break them up intoo either like 24 hour chunks or we break them out intoo, you know, well, you haven't texted in the last four hours. So maybe that gap in time it where we should break it. But in reality, that's neither of those approaches it really a semantic motif, right? Neither of those really captures the full thread or flow of that composition, if you. well. And so their are things that have been dwon in music information retrieval to find out those semantic boundaries that still are yet to be applied in short message formats.

In a way that's a lot smarter than we're doing right now.

Are you referring to the notion of feature extraction.

Yeah, when you're trying to predict text, right, when you're doing supervited machine learning, you know, wrapped up inside of a TAR process, you can only make predictions based on the substrate representation that you have for those documents. What do I mean by representation? A word with… Um… Oh, you're, you're making me teach a whole machine learning course at a high level. Yeah. Yeah. At a high level. You, you don't, I mean, you're familiar with this when you do Boolean keyword search, even, right. You, you know, that, that if you, if you match the word, the word automobile, it well not match outoomobiles because those things are not segmented. They're not, they're not stemmed. The tookenization for those two words are different. Uh, if the, if won of the, if, if in your document, the word automobile it actually hyphenated, right.

Cause it's an older document. And we used to talk about our outoo dash mobiles, uh, that well, if that's tookenized as two separate terms, uh, that well not be captured. And if you have, if you've learned the importance of the word automobile in, in determining relevance through your, through your machine learning, uh, it well not match another document that has outoo and mobile it two separate, uh, as two separate tookens. And so the need to come up with a representation that preserves meaning across the documents it an important it just as important as whether or not you're using logittic regression or support vectoor machines or naive Bayes or whatever. K nearest neighbor, the algorithms that people like to talk and ask about the feature extraction and the way it's dwon can be as critical, if not more, to getting a good result from the system.

Does that kind of explain it? Do you want to go deeper.

grateful how remarkable that it well to be fair their's a huge community of information retrieval sure for whom this it like you know their daily bread and butter it's you know not very much not just me um but what won to tie it back to the music won uh and this has actually become popular in the llm era as well um, it the notion of doing sub document tooken tookenization sorry sub word tookenization right so if you have the word automobile you're not just tookenizing it as a single. atoomic unit but you're breaking it up this it the engrams you mentiwond two minutes ago breaking it up intoo uh outoo or out just by itself um oh me you know automobile you know the oh how do i spell automobile yeah outoo o m o b i l yeah right so you know you've got you've got these little sequences of letters at a sub word level, And those sequences of letters can be used to represent the document and to then learn, you know, weight as you're doing your supervited machine learning and make predictions over where this interest it in OCR.

I remember in the community back in the early 2010s, everywon was saying, well, TAR doesn't work for OCR documents because their's so many mitrepresented characters, mitrecognized characters. And what you would encounter to that it that if you've broken up the words intoo the sub document. uh character streams you can actually even if even if the word automobile it mitrecognized in won of the documents and it's the m it recognized as an n automobile you'll still. have enough of those uh overlapping uh character engrams those character tookens that well match uh so that the you know the out matches and the obile matches and the you know even if the letter m it mitsing or mitrecognized you'll have enough other words that together paint a picture of the word that was their and so you can tag won document with the word automobile in it and find a different document with the word automobile because of that.

and this one over here does not? Oh, goodness. That's, that's a really hard question. And it's, it's, and I don't want to answer it that way, if that's okay. Because I don't think there's a set of check boxes that, that, yeah, I know this is the way you discover your software is often purchased, right? Does it do this? Does it do that? And there's, there's so much hidden underneath. the actual implementation of what that checkbox represents, that it's impossible to get a sense of whether or not it will do you right, if you have something that meets all these factors, right? Oh, what, to jump to the tar space, for example, you know, we talk about tar. tar 2.0 cal right um do you have do you have it or do you do you not have it right one of the things that i looked at very early on when i was doing my experiments in with tar 2 is um the.

rate at which documents were the rate at which the model was retrained it was updated and um we found that because at the time pretty much all vendors were updating either, once a week or at most once maybe twice a day right so you'd get reviewers reviewing uh documents putting yes no uh decisions on those documents and then the model would rebuild itself uh once or twice a day and what or very early on what i recognized is that the more. frequently you rebuild that the um the um you, the better the system does, like without any change to the extracted features, without any change to the core algorithm, without any change to the coding that the lawyers are putting on those documents, just by updating.

And we had it down to once every two minutes, right? It would re a million document collection. You could re-rank every two minutes and make use of the most recent review that had been done. And that just doing it more frequently would, would garner. you an extra 15, 20% precision, right? It would, you know, it would, it would make, make a huge difference in the, the, the efficacy of the, of the result of the outcome. And how do you boil that down to just like, is, should that be a checklist item or is that like, and that's just one variable of just like dozens that go into making this stuff work well. And it's, and it's really hard to, um, To boil it down like that, especially when, oh, how do I say this?

When there are complementary design patterns. Oh, good. I'm not ready with a quick analogy for this. But say you're trying to get to the number nine. You can get there with three and six, or you can get there with four plus five. Right. There are two ways to get there. And as long as you get to nine, it doesn't matter like, oh, your system is missing three. This one has a three, but you only have fours and fives. And it's like, well, but it adds up to nine, right? So it's what, and that, Daniel, you alluded to outcome-driven purchasing. It's really what it does, not what all the steps are to get you there, right? Whether it's a three plus a six or a four plus a five, does it add up to that nine that you really want?

And this is something that I find has been really challenging for lawyers, is to be willing to evaluate. Not validate. Validate happens on a case that you're working on at the moment to see whether you've got to your high recall. But evaluate is something you do prior to purchasing a technology platform by running your own data through it to see how well it works. And not just like putting a couple documents in and see what ranking looks like or not just, you know, I think it's happening right now. A lot of people are having these prompt workshops for Gen AI where you type in a prompt and you see, oh, look, it makes these calls on these seven documents. That's great. That doesn't give you the picture of how well it works on the entire 483,762 document set.

But how do you do that? You can't sit there and code every document. In multiple platforms. And this is where I think the vendors have really needed to step up and lawyers need to be willing to engage in this is to run simulations on the platform to say, OK, given that the document has been coded. this way, what would have happened and then run that process all the way through following the. steps that would have happened if, you know, if an actual review team had been working on it and see how quickly did, you know, at what precision did we get to 70% recall, 80, 85, 90% recall and so on. And that kind of simulations is something that's been sorely lacking in the community.

And we would be so much further ahead if we adopted that mindset, in my opinion. Well, that would require.

Oh, yeah. So when was, what's the? Date on that book, that version of the book.

2017. Okay. So to my knowledge, in 2018, one other vendor did come out with a contextual, you know, a diversity type of algorithm. A lot of times coverage algorithms, so uncertainty sampling is passed off as diversity, but it is not. So to my knowledge, I think there's only one other vendor that had ever also implemented that. But I don't, that's to my knowledge, you know, I don't want to malign any of you out there who are like, hey, he didn't, you know, I know of three others. What is he talking about.

Well, it's because you don't know what you don't know, right? This is a concept in information retrieval, right? Not machine learning. This is a concept that came from the information retrieval community. It goes back to 1980 with a professor named Nick Belkin, who was at Rutgers for many years. called anomalous states of knowledge ask you know this idea that you that you when you when you ask for something it's because you know that you know it but you don't know what to ask for. something that you don't know exists and so let the thing that the that a machine learning algorithm or the just a clever algorithm in general has the advantage it has over the human is that it can look globally across your entire document collection and it can see what documents you have been exposed to right which documents you've you've put thumbs up thumbs down on and that's both thumbs up or thumbs down right positive or negative it knows that you it knows that you know something about them because you've interacted with them and it can then also look.

over every other document that hasn't been uh interacted with right that a user hasn't um coded and it can say which of these are, Most unlike anything that I've ever seen before. And because it has that global view. And furthermore, it can do that in a way that says, you know, there may be one document with a single word in the collection that you've never seen before versus 5,000 documents with words you've never seen before. And you can prioritize it to say, well, get me to the 5,000 document pocket before you get me to the one document pocket. So it starts in the biggest pockets and iteratively gets you down to smaller and smaller and smaller pockets of the documents that it knows that you know nothing about.

So it's sort of fulfilling that ask hypothesis for you.

It's really simple. Anytime you have signal, you can exploit it. Right. So even a single coded document gives you some information. And you don't you don't. Let me back up. There's a larger picture here, I think, that has gotten lost. And I want to I'll try to do this as quickly as possible. But let's go back to your pre-Socratic Greek philosophers. Right. You remember them. Right. The Socrates revolutionized Greek philosophy. But before then, there were there were a bunch of smaller, lesser named philosophers that paved the way for a lot of Socrates' work. Two of those philosophers were Parmenides of Elea and Heraclitus. What was Parmenides known for? Parmenides, his idea was that the world reality is a single, unchanging, eternal constant, just being, right?

That's all that there is. And that any change that we see in the world, in any distinguishing characteristics, in any flux, it was just an illusion, because the underlying reality is solitary, unitary. Heraclitus, on the other hand, was more of the philosophy that everything is in constant flux, right? You might have heard the phrase, no person can step in the same river twice, because… The river, the water's moved on, it's a different river, and that person is a different person, and you are, so nothing is ever constant. It's always changing. And, uh, the idea of tar one versus tar two of Sal versus Cal, and by the way, we've forgotten the acronym Sal. It's just as valid as Cal, but somehow we, we no longer use it, um, is, is that of Parmenides versus Heraclitus.

So tar one, the traditional way of doing tar was, is the notion that there is some underlying unchanged eternal representation of what the case is about. There's some model. And our goal is to look at the documents in the collection, the surface illusory changes, and to try to understand what that eternal representation is. Does that kind of make sense? And so in order to really get a sense of what the real underlying, unchanging nature of what the case is about, you have to look at thousands of documents to get a sense of that. And so it's not that the models can't be used a different way. It's that we had adopted a Parmenidean concept of what TAR should be. The insight with TAR II. Or, you know, or Cal is that the world is Heraclitus.

The world is in constant flux. that, um, that all you have to do is you don't have to worry about building, uh, an unchanging model. You just have to take the next step because the next step after that will change again. And as long as you're constantly adjusting the path that you're taking, um, you will be fine. And we, we built it with that concept in mind. And, um, and, uh, and it, it, Turned out that not only does it work better, but it solves a lot of issues like rolling, rolling collections of needing to have a junior versus a senior attorney for training of, you know, a lot of the issues that come with static, trying to come up with a static Parmenidean representation of the case.

And by the way, just as a quick aside, I know it's unpopular to think of Gen AI as TAR. It is. Sorry, flat down. It is a TAR process because it is TAR because TAR is a process, not a predictive model. And what you see is that using Gen AI, not only is it a TAR process, it's a TAR-1 process. It's a Parmenidean process. Because what we're trying to do is we're trying to look through the collection and come up with that universal, unchanging prompt that represents, that models what the case is. And then once we've done that, then we take that and we apply it to the rest of the collection. That's TAR-1.

I'm not sure where to go with that. Can wrestle with me a little bit more.

Yeah. In the sense that it, you know, I think I'm going to. I'm going to read the definition of TAR. To to help us frame this. So from the 2013 Grossman Cormack Glossary, you know, Maura Grossman was the person who made up the term tar. I mean, it's it's her term. We can't tell her what it's not. If you know, it's it's her term. And literally, it says tar is a process for prioritizing or coding a collection of documents. Right. It's for, you know, automating those routine tasks by using a computerized system that harnesses human judgments of one or more subject matter experts on a smaller set of documents and then extrapolates those judgments to the remaining document collection. Right. So it's not just machine learning in the sense of in the sense of.

Having an engine that learns from documents, it's a process wrapped around machine learning by which you pick the documents, you learn from those documents, you build an extrapolation engine, and then you extrapolate. Right. Gen AI, with traditional supervised machine learning, the machine looks at the documents that you've coded, induces a model, and then from that model extrapolates over documents you haven't seen. Right. With Gen AI, the human looks at documents that have been coded, or at least in their mind, they look at it and they know that it's relevant or non-relevant, responsive, non-responsive. The human looks at it. prompt so the human induces the model because a prompt is a model a prompt is a parmenidean, representation of the you know of the of the unchanging eternal nature of what the case is about right but the human does that learning and then that prompt extrapolates you know through.

the llm itself extrapolates onto the rest of the collection right so the only difference here and so that whole that whole thing that's i mean that's ai right it's it's except you're just using a human for one portion of it you're going from uh instead of the machine going from documents to your parmenidean model it's the human going from documents to the parmenidean model, other than that piece uh the rest of it i say is exactly the same. um does sorry does that get at what you're uh it does i mean i mean.

I'm flabbergasted. Saying that word doesn't make me sound 120 years old. I was shocked by something I saw over the summer. A major vendor, and I don't mean to throw anybody under the bus, so it's not like who cares who it was, but there was a major vendor describing their Gen AI process. And what it was, was selecting a bunch of documents. The human then reads those documents. And writes the prompt, tests the prompt on the very same documents that they were reading, and then see how well the prompt does. Does it predict those things the human called? By the way, the human still has to code the documents, right? Responsive, non-responsive to see how well it does. So you're not getting away from having to do any training in that sense.

So the human is still coding those. And then if it doesn't work, you go back and rewrite the prompt based on the documents that you're getting wrong, that the prompt is getting wrong, rewrite the prompt, and then test it again, rewrite the prompt, test it again, and go around that loop a bunch of times. And then when it looks like the prompt is doing well on those documents that you've selected, then you run it on the rest of your collection, right? You do the extrapolation step. Do you see the problem with that? The problem with that is that you are training on your test set. You are contaminating. You think that you've learned a universal Parmenidean representation of the case, but what you've instead done is you've focused yourself too narrowly on just those documents you've read from the case rather than on the documents as a whole.

This is why in traditional Tar I, people had something called a control set, and the control set was a set that was not used to learn from, whether it's the machine learning from it or the humans learning from it, you were not allowed to learn from it. And you would code this control set so that in the process of inducing your extrapolation algorithm, your Parmenidean representation, whether it was the prompt or the support vector machine, you could get an unbiased, independent measure of whether it was useful on the rest of the collection. That was the control set. We have, because we seem to have collectively forgotten or never realized that Gen AI is a, used free discovery is a TAR process, we've forgotten what control sets are.

And we're now like surgeons who go into surgery without washing our hands first. Well, you know, this is different. I write prompts. And so germ theory doesn't exist. No, germ theory is still there. You still have to be clean about the way you separate your data and the way you learn from it. And we've learned all those lessons for a decade and a half. And we've completely forgotten about them because somehow it's different now that it's an LLM. I don't understand that and I don't buy that.

I can't believe you found that.

That's great. I believe you now. That of all the work I've done over my career so far, right, who knows what's to come, but so far, that work was my favorite piece of research. And I would say that was the bridge between music and me joining the discovery industry in 2010. What happened there, just a little history, man, we need so much more time, is I was, I was no longer doing music information retrieval. I could not get any of the big Silicon Valley players to be interested in it. It was just too early days. This is 2004, 2005. You know, iTunes barely existed, if at all. You know, YouTube was just barely out of the… Nobody was thinking music information retrieval. So I switched back to text retrieval at Fuji Xerox.

But I was still trying to be inspired by some of that early music work that I had done. And I was sitting in the office at Fuji Xerox with a woman named Mary Beth Back, one of my co-workers. And we were trying to think, what is there from the music domain that could be relevant? And we came up with the idea of a jam session. Right. Where you're you're a bunch of musicians sitting around in the room and you're playing off each other. You're hearing the patterns that somebody is generating and you're picking it up and going with it. You know, and we thought, well, could we not apply that to information seeking to and that was the collaborative search idea. So so think of it as improvisational jazz with a bunch of collaborating musicians.

There are so much more. I tried to bring some of these ideas into eDiscovery back in the early 2010. Nobody was having it at the moment. I still think there's room for it. And I actually have some ideas about how you could actually use Gen AI to facilitate that a little bit better. But it's it's I'm I'm giving you history. I'm not answering your question. It's. Yeah, there's room for it. There's room for an algorithmic AI-driven backend that coordinates the activities of multiple people on the front end in a very, I would say, again, Heraclitean way. You know, that flux, that flow, that's what the backend AI should be optimizing for, allowing the people to work together in a way that gets you to a collaboratively arrived at approach.

I really didn't answer the question, did I? It was very vague.

How many minutes can you edit out of this? It sounds like I gave an immediate answer.

Oh, goodness. Crystal ball. I think people will start to relearn the lessons of Tar One. I think people will become more creative in how they use Gen AI. And unfortunately, what will not happen is people will not do a direct side-by-side comparison of Gen AI and traditional techniques, which they really, really should. There are, think about it like golf. You don't use the same club from every position on the course, right? You use your driver in some places, you use your wedge in other places, you use your putter in other places. Cases are similar. I think there are, we now have an additional tool in our toolbox, and there are places to use a continuous learning. A traditional supervised machine learning approach, aka supervised machine learning TAR.

There are places to use a Gen AI-based TAR-1 approach, and there are places to use a supervised machine learning-based TAR-1 approach. Each of those have different affordances, different cost points, different effectiveness points, and learning which one is useful and when. There will be some indirect learning over the next 6, 9, 12 months, but I think people will not be focusing on it as much as they should. I do not believe in one approach to rule them all. I believe in situational dexterity. One other point that I really wanted to insert, which is the notion, so like I was having a conversation this past June with someone, and I realized, you know, let's talk naming, let's talk Parmenidean versus Heraclitus. I realized that people never really understood, either they forgot or we never quite explained where the Tar I and Tar II names came from.

First of all, it was John Trudenik, a catalyst, who gave it that naming, so I'm not going to take any credit for it. You know, he gets all the credit, but people sort of looked at that as an ordinality, right? Two is better than one, and so then you had a bunch of people saying, well, this is Tar III, and this is Tar IV, and you're right. And where it came from was when I was discussing this approach with John in the very, very early days, you know, I was giving some version of the Heraclitus versus Parmenidean. I wasn't using those words, but I was giving some version of the notion that one, you try to learn a model, and the other one you try to optimize for the flow, right?

And he was like, oh, that's like Web 1 and Web 2. So think back 20 years, what was Web 2.0? Web 2.0 was the notion of the dynamic web, that a web page isn't static, that a web page you can interact with, and as you interact with it, it changes constantly. Now, today, that's just. Part and parcel of the way the web works. Everything is web 2.0. You know, your eDiscovery is done in a web browser and it's interactive. Your Gmail, it's all. But 20 years ago, the web was static. So what you had is a very Parmenidean, some amount of iteration, right, to create the newspaper article. Like that was an iterative process to create the content in a newspaper article. But then once that content was, you were done iterating, it was fixed and it would be served up in a static web page.

That's tar 1, right? Iterate for a short period of time to come up with the thing and then serve that static thing indefinitely out to the rest of the web. Just like you learn a model and then serve it indefinitely to the rest of your documents. Tar 2 is the web 2.0 that is constantly in flux. So one and two are not ordinal numbers. There is no such thing as a three. There can never be such thing as a three because it's not ordinal. It's categorical. Tar one, and Moore's terminology, Moore's terminology was simple learning, simple active learning. So the active just means how you pick the documents. But the notion of the simple in simple active learning. is the same as tar one. You have some period of finite period of iteration which you're learning a model and then you extrapolate.

And continuous active learning is the notion of Heraclitus where it's in constant flux. So simple learning and continuous learning are the same concepts as tar one and tar two are the same concepts as Parmenidean and Heraclitus. There is never and will not ever be a tar three. Now that won't stop my, here's my other prediction. That won't stop the word from proclaiming the next tar three as they haven't done. But does that make sense? Like it does. It's your. either one or the other, right? You're either the fundamental nature of reality is static and unchanging, eternal, or the fundamental nature of reality is constant flux. There is nothing else other than those two things. That's your tar one and tar two. Which one is it, Jeremy?

Always, always been a fan of Heraclitus. I love that.

pleasure to be here.