A Reply to West Publishing Company and Mead Data Central on The Curse of Thamus.*



I am pleased that "The Curse of Thamus".1 has attracted the attention of West Publishing Company and Mead Data Central, two electronic publishers whose computer-assisted legal research (CALR) systems are discussed in the article. It is hardly surprising that West and Mead do not entirely agree with my analysis. "The Curse of Thamus" speaks for itself on most of the points addressed by West and Mead, but on a few matters some additional comments may be helpful.
Both publishers argue that the results of the experiment performed by Blair and Maron 2 on a litigation support data base should not be generalized to all full-text document retrieval systems, or at least not to LEXIS and WESTLAW. Preliminary data on the retrieval effectiveness of LEXIS and WESTLAW themselves, however, tend to confirm the findings of Blair and Maron. A pilot study by Scott Williams3 yielded the following results:
Recall Confidence Precision Confidence
(95%) (95%)
LEXIS 11.4% + /- 5.4% 26.1% + /- 7.2%
WESTLAW 19.7% + /-6.8% 26.9% + /- 5.8%

Mead refers to a study by the Legal Services Corporation4 as contradicting, the results obtained by Blair and Maron, but in fact the two studies are consistent. The Legal Services Corporation study did not measure the Performance of full-text document retrieval in terms of recall and precision, but rather in terms of user satisfaction, Blair and Maron found that users thought they were getting at least 75 percent recall when they were really getting no more than 20 percent.5 This is entirely consistent with the Legal Services Corporation's finding that users thought computer-assisted legal research systems were giving them reasonably thorough research. 6
West takes a more generous view than "The Curse of Thamus" does of the helpfulness of the editorial apparatus added to WESTLAW cases. The argument is plausible, but West overstates the empirical support for such a conclusion. The article by Al Coco 7 showed not that WESTLAW gives higher recall than LEXIS, but that WESTLAW retrieves more documents than LEXIS does, This does not translate into an improvement in recall unless it also can be shown that precision among the additional documents is better than precision for the search as a whole. Carol Tenopir's data show this to be true for the Harvard Business Review data base, 8but it might not be true generally. The data base in Tenopir's study contained only about 900 document, and, as discussed in "The Curse of Thamus", the retrieval characteristics of small data bases differ greatly from those of huge systems like WESTLAW.
The work done by Blair and Maron casts doubt on the ability of full-text document retrieval systems like LEXIS and WESTLAW to provide users with high recall key-word subject access to their data bases. "The Curse of Thornbush" examines the results of Blair and Maron in the context of CALR. The study by Williams, the preliminary results of which are reported above, tends to confirm that the results Blair and Maron are applicable to CALR.
Much can be done to enhance the retrieval performance of LEXIS and WESTLAW, but many of the possible improvements would be expensive to implement. If there is to be improvement in commercially available CALR systems, users must be aware of the problems and bring pressure to bear. CALR publishers have been responsive to user needs in the past, and we can expect that their products will continue to improve.

*© Daniel P, Dabney. 1986.
**Head of the Reference Center, University of New Mexico Law Library, Albuquerque, New Mexico.
1. Dabney, The Curse of Thamus: An Analysis of full-Text Legal Document Retrieval,78 Law Libr. J. S. (1986).
2. Blair & Maron, An Evaluation of Retrieval Effectiveness for a Full-Text Document Retrieval System, 28 Com. A.C.M. 289 (1985) (publication of the Association for Computing Machinery).
3.This data is from research in progress at the University of Texas School of Law.
4.Legal Services Corporation, Final Evaluation Report, Computer-Assisted Legal Research and Technological Improvements(1981).
5. Blair & Maron supra note 2, at 293
6.Legal Services Corporation, supra note 4, at 104. 7.COCO, Full-Text vs. Full-Text Plus Editorial Additions: Comparative Retrieval Effectiveness of the Lexis and WESTLAW systems,Legal Reference Services Q., Summer 1984 at 27.
8. See C. Tenopir, Retrieval Performance in a Full-Text Journal Article Database(1984). West cited an article that discusses parts of the same research. See Tenopir, Contributions of Value Added Fields and Full-Text Searching in Full-Text databases, in proceedings of the Sixth National Online Meeting 463 (1985)


Page maintained by YLS Computer Services Web Team, and page URL(http://www.law.yale.edu/lawweb/lawcrs/lasdab2.htm) [ Go to Top of Page ] [ Go back to Menu ] [ Yale Law School Home Page ]