The SRC Report: Decoding Legal Research – Can AI Take the Stand?

Welcome, everybody, to a special edition of the SRC Report. I'm Athena Cains, principal at McInnes Wilson Lawyers, and I'm joined today by Sarid Milne, one of our lawyers in the Commonwealth Compensation team. Thank you for joining us, Sarid.

Thank you so much, Athena. Lovely to be here.

Sarid and I wanted to talk today about the use of artificial intelligence in conducting legal research. And although that's probably not a topic that we would usually cover in the SRC Report, which is certainly aimed at HR professionals managing employees under the SRC Act, we did think it was an incredibly interesting topic given the outcome of the research project that Sarid undertook recently for us.

So one of the things I think everybody has heard of ChatGPT, and now we're aware of Google Bard, which is a more recent AI, artificial intelligence, tool to generate, I guess, documents. And I think our students might be using it for their research assignments and all sorts of things. And I think us, along with probably many professional services firms, thought it might be quite useful in assisting us to conduct legal research and really narrowing down to relevant cases and relevant law on what can be sometimes very arcane areas of law that we're asked to advise on.

So Sarid undertook that task for us in our team. I wanted Sarid today to just talk us through what happened when he threw a recent federal court decision at ChatGPT and Google Bard, what we got out of it. So Sarid, can you just take us to what the case was, and how you went about your project?

Absolutely. So obviously, we're all aware that AI is quickly, rapidly developing. And I think there's a real excitement to want it to be exactly what you want it to be, that it can spit you out the answer incredibly quickly. It's definitely going to be something that will be present in the future. But based on our results, it might not be here just yet.

Not ready yet.

No, not quite. So we took a look at the case of McEwan and Comcare. This is a 2023 Federal Court of Australia matter. So I'm not sure whether it be a fun game to see if you can pick which one was the AI's summary and which one was the actual case. So we put in the case with a couple of different prompts with Googlebot and ChatGPT. One of the first things to note with ChatGPT was that it quickly advised that it doesn't have access to the internet. So it's not able to review up-to-date websites. So it quickly said, "No, no. I can't help you here. Recommend that you go to a lawyer." Always very nice to hear. Usually, we don't get those kind of referrals, but good to see that AI is on our side for now.

For now.

For now. And then, I moved across to testing it out with Google Bard, and it had some very interesting results. So firstly, I tried quick request to see if it could provide a summary, providing it with a full citation of the decision. And it proceeded to spit out a case nodes with all of these beautiful headings and paragraphs beneath it. But I quickly realized that what it was actually providing me was it was something that is entirely different to what the actual case was.

So the case of McEwan and Comcare is in relation to an appeal from the administrative tribunal where the applicant sought a review of the decision where Comcare had denied liability for a depressive disorder and generalized anxiety disorder caused by systematic bullying. So that was the actual case, and that can be contrasted, I guess, with what Google Bard spat out, which was that this was a case involving a traumatic brain injury sustained as a result of a motor vehicle accident.

So already, we can see that they're pretty marked differences there. Not at all the same. And quite plainly, it's just factually incorrect. I guess this is kind of already from the get go highlighting the dangers of relying on an AI without actually reading the source material yourself.

But we had a play around with it. With this test, I had to look through the case myself first just to make sure that I was across the relevant details at a surface level. I saw that I could identify when these issues cropped up. And like I was saying before, very quickly apparent that what it provided was not at all factually correct. What shows up with the chat response does look like it is a real case. It gives you a judge. It gives you a date of a decision, the background to the info to the case, the argument that was supposedly put before the court, and the salient reasoning of the judge here.

But the difficulty here or the issue is that that none of this actually occurred. It's all wrong. The argument the were put forward, the judge isn't the same judge. The date of the decision isn't the date of the decision. So it's quite fascinating that it's managed to completely create something from nothing, even though it's being pointed in the exact right direction of where to look.

And I tested this a couple of times with a few different prompts. And for whatever reason I kept getting the same response. So this was including giving it the link to access to say, "This is what you need to look at. Please, summarize this based on this," and got the same thing. So it really seemed quite fixated on this hypothetical motor-

Fictional case.

... vehicle accident. Yeah. I was trying to find it, but I couldn't find anything or at least nothing recently.

Wow. But then you took matters into your own hands and tried a different way, didn't you?

That's correct. So I thought, "Okay, it seems to struggle with references to certain materials." What happens if I actually give it the entire case? Now, a lot of the chatbots seem to have a cap on the amount of characters or words that you can provide to it. So ChatGPT seems to have a pretty short character limit. Google Bard, when I'm using it on the desktop, some reason seems to have also a fairly short character limit. But quite strangely, when I used it on my phone to do the same task, I was able to put in the entire judgment into the prompt box and ask for a summary based on that. And it did actually spit out a summary, and it was actually finally in relation to the real case. And so what it was spitting out was decent, I would call it. It wasn't by any means a comprehensive analysis of the case.

It was correct in terms of the background materials, the background facts at a high level, not anything specific, but it did provide a rough outline of the case, the previous decision made in the tribunal and the real cause of the appeal. The difference though is that the chatbot has reached a conclusion that if you're a trained legal practitioner or someone that's familiar with the case, it's a different conclusion to what was actually provided by the judge in the case.

So Google Bard has, I guess, summarized this decision as being that the... I should give some context here. The primary reason the ground of appeal in this case was whether the tribunal had sufficiently considered where the aggravation of the injury had occurred. This wasn't strictly brought before the tribunal's attention, but the appellant argued that it was clear on the evidence that that was something that should have been considered.

The court ultimately held that wasn't the case. This wasn't the matter that was run before the tribunal, and that wasn't made out in the case run by the applicant. The chatbot, on the other hand, it concluded that the applicant hadn't proven that they had a depressive disorder and generalized anxiety disorder and therefore the tribunal wasn't required to consider aggravation.

So it doesn't quite get to the correct conclusion. And that seems to be a bit of an issue because the real important part of that case was what the tribunal needs to consider when a case is provided to it. There was a consideration of the case of Ellison and Comcare, which is a pretty important decision in this field. And actually the court distinguished that case and provided some commentary in relation to it, which is probably the most important outcome from this decision. But that's not something that the chatbot was able to articulate or to provide.

So it demonstrates that even though it does have all of the relevant materials in front of it, it's not quite able to make those legal conclusions or assess things it gets from a lateral thinking method or see what the broader ramifications might be, which I think as we were talking about right at the start of this, is probably the biggest risk with relying on or thinking that chatbots are maybe better than they are, is that they can spit out something that looks quite good. But when you actually dig deeper into it, it's not actually what reality is. So my view probably not quite here yet is the AI.

Which is both good news for a lawyer, and also bad news for a lawyer. I have to admit, I've been using it a little bit recently just to test it out, not in quite the same way as Sarid. But I've been using it to give me a bit of a steer on some complex concepts of administrative law that are really hard to, I guess, distill down into a search into AustLII. And I'd have to say it's reasonably good. So I was looking at the difference between delegation and authorization.

It wasn't completely wrong, but I think where it really lets me down as a legal researcher and advisor is it might provide you with an answer, but it doesn't provide you with the source material. I mean, it's obviously a bit easier when you give them the key case that you're looking for, but to drill down into the concepts it describes is very difficult from the product that it produces, which means that it may help you. Potentially so far, it's helped me refine some search terms, I would say, but hasn't actually really sped up my process much more than that.

So I'll be interested to see whether that's something that's a refinement or whether there's going to be somebody, and surely there's going to be somebody any minute now who will release an AI in relation to legal research. But if I was a law student, I would not be relying on ChatGPT or Google Bard to write my essay.

No, not at all.

Definitely not out of our research.

Absolutely. And I think more as well, particularly for people that aren't even in the legal sphere that are trying to keep up with case law, and maybe it's cases that affect you, for example, with this, if you're an HR employee and dealing with the SRC Act on a regular basis, you might be wanting to try and use something like this to just summarize the relevant decisions you might've heard. There was a great case that came out over last week and got a really important decision, but it's a couple hundred pages long. You don't want to read that.

No

You might think, "I can do that..."

Nobody wants to read it.

Exactly.

Yeah. But don't do that.

Exactly. What we found is that it's not the way to do it, or at least at the moment, we'll see what comes in the future, but currently not to be relied on in any circumstances for a summary of a case.

Yeah. Well, thank you so much, Sarid. That was a really fascinating project that we were involved in. I really hope that at some point the AI does improve to a point where it can be relied upon because, gosh, that will make life easier for all of us, but it's just really interesting outcomes there, and I guess some salient warnings too about using AI at this point in relation to legal research. So thank you so much for joining me today.

Not a problem. Thank you so much for having me.

Oh, anytime. So thanks everybody for joining us today to talk about the application of AI in legal research. As I think I described on every SRC Report, please do consider subscribing to our LawFlix. It's McInnes Wilson's bespoke searchable CPD content, and contains lots of really interesting legal seminars that will be of relevance to almost everybody who listens to this. So thank you, Sarid. And thank you, everybody. I'll speak to you again soon.