Using Situation Theory to evaluate XML Retrieval

1. Auflage 2012
ISBN: 978-3-89838-506-0
Verlag: Akademische Verlagsgesellschaft AKA

Buch, Englisch, Band 106, 238 Seiten, PB, Format (B × H): 148 mm x 210 mm

Reihe: Dissertationen zu Datenbanken und Informationssystemen

This book presents a theoretical framework to evaluate XML retrieval using Situation Theory. XML retrieval deals with retrieving those document components, the XML elements, that specifically answer a query. Theoretical evaluation is concerned with the formal representation of qualitative properties of retrieval models. It complements experimental methods by showing the properties of the underlying reasoning assumptions that decide when a document is about a query. We define a theoretical methodology based on the idea of ‘aboutness’ and Situation Theory and apply it to current XML retrieval models. This allows comparing and analyzing the reasoning behaviour of XML retrieval models. For each model we derive functional and qualitative properties that qualify its formal behaviour. We then use these properties to explain experimental results. We further analyse specific characteristics of XML retrieval such as the attempt to only retrieval the most focussed answers as well as specific evaluation criteria.

Zielgruppe

Universities, Societies, Research Libraries

Autoren/Hrsg.

Blanke, Tobias

Leseproben

Chapter 1
Introduction
In information retrieval (IR), we want to find information sources,which can be documents or more complex ones such as multimediafiles, that are relevant to an information need. For IR, this informationneed is expressed in a user query. This computational task tomatch a user’s information need to information sources is non-trivialand implies complex artificial intelligence techniques and statistics.XML retrieval adds to this complexity by considering semi-structureddocuments, written in XML. XML, contrary to HTML, separates thelogical structure of documents from the layout. XML retrieval is concernedwith using this document structure to improve the retrievalof information from documents by only delivering those parts of adocument that are the most relevant ones to an information need.Given that XML has become an accepted standard method tostructure information, making it easier for applications and devicesof all kinds to use and store information, XML retrieval has also becomean increasingly important research topic. This work attempts atheoretical evaluation of XML retrieval and develops a general frameworkto do so.Theoretical evaluation of information retrieval is concerned withthe formal representation of retrieval models, which includes the symbolicrepresentation of the way a retrieval model captures informationand the analysis of the matching function between information needand document. A theoretical evaluation is then complementary toan experimental evaluation if it helps to clarify the assumptions ofretrieval models and if it can identify the characteristics leading to aparticular experimental behaviour.2 Chapter 1. IntroductionIn this work, we define a theoretical methodology based on theidea of ‘aboutness’ and apply it to XML retrieval models. Our frameworkto evaluate XML retrieval is based on the basic and most generalinformation retrieval question: how a document (in our case an XMLstructuredone) can be about a query. This allows us to compare andanalyse the behaviour of XML retrieval models.As we consider aboutness as the cornerstone of our theoreticalevaluation, we need a way of expressing aboutness relations. To thisend, our work draws on existing logic-based approaches in order toderive the properties of aboutness relations and to analyse informationretrieval processes. It will become clear in our work that anaboutness-based theoretical evaluation can complement mainstreamexperimental evaluations and provides an opportunity to produce newand redevelop traditional IR models. This is especially important forthe more complex IR tasks of the future, one of which is the effectiveretrieval from XML document repositories.Our general hypothesis is that particularly in the domain of XMLretrieval, an aboutness-based theoretical evaluation presents a powerfulmethodology to analyse the complex interaction of XML structureand content. This work will adjust existing aboutness-based evaluationapproaches to reflect the requirements of XML retrieval anddevelop a new methodology. Based on this new methodology we willbe able to theoretically evaluate various XML retrieval models andunderlying aboutness assumptions of specifics of XML retrieval suchas experimental evaluation strategies. We will argue that our theoreticalevaluation leads to a better understanding of a model’s retrievalperformance in the experimental evaluation.This work is organised in three larger parts. The first one introducesthe background, the second develops our methodology, whilethe third applies the methodology to analyse XML retrieval modelsand further specifics of XML retrieval such as its experimentalevaluation strategies.Chapter 2 is the first of two background chapters introducing XMLretrieval in more depth and our choice for a theoretical evaluationframework. It introduces basic XML concepts relevant to this workand finally describes the subdiscipline of information retrieval calledXML retrieval. It details some of the history of XML retrieval as wellas XML retrieval approaches that are emerging in this field. Furthermore,we discuss in detail evaluation methodologies that have beenChapter 1. Introduction 3developed to support the specific aims of XML retrieval in the contextof specific evaluation initiatives.While the analysis of XML retrieval is well developed by now,there has been little, if any, work done on a systematic theoreticalevaluation. This work undertakes such a theoretical evaluation. Forthis purpose, the second background Chapter 3 introduces the conceptof theoretical evaluation, its history, predominant approachesand finally our choice of using an aboutness-based approach to theoreticallyanalyse XML retrieval. We demonstrate why a theoreticalevaluation based on aboutness is needed, as it helps reveal some ofthe underlying assumptions of the XML retrieval work done so far.In Chapter 3, we also introduce the framework we need to executeour theoretical evaluation based on the logical analysis of reasoningprocesses involved in XML retrieval models. We build uponan advanced mathematisation of natural language semantics calledSituation Theory. We introduce its basic concepts and show howits ontology is particularly well suited for the analysis of informationretrieval models. We derive what an aboutness analysis based on SituationTheory means for XML retrieval in particular. We introduceXML structure to aboutness research, as a component of the aboutnessdecision and develop a framework for this, so that at the endof this chapter we are able to generally define XML retrieval aboutnessand redefine existing approaches to Situation Theory aboutness,which allow for the inclusion of structure.While Chapters 2 and 3 offer the background, Chapter 4 presentsfirst results of our work. Our research into developing an aboutnessbasedtheoretical evaluation of XML retrieval commences with thedevelopment of our methodology, where we adjust existing theoreticalevaluation methodologies to the requirements of XML retrieval,analyse gaps and finally amend existing approaches with new partsthat help describe the reasoning behaviour of XML retrieval models.We develop our theoretical evaluation steps and present with puretype XML retrieval a means to measure the impact of XML structureon the reasoning behaviour of XML retrieval models.Our evaluation methodology stands in the tradition of theoreticalevaluation as it uses a well-defined number of steps to firstly symbolicallyrepresent an XML retrieval model and to secondly analyse itsfunctional behaviour using logical reasoning rules. Each of our theoreticalevaluations of XML retrieval models goes through the same4 Chapter 1. Introductionfour steps to define the characteristics of a particular XML retrievalmodel.The first step is the translation of the way the model indexes informationinto a symbolic representation, which we can use in thesecond step to analyse the aboutness definition with reasoning rules.In Section 4.4, we define reasoning rules to describe the functional behaviourof XML retrieval models. Our discussion of aboutness rulescovers basic rules, then combination and containment rules beforewe finish by discussing how non-aboutness reasoning can enhance IRmodels. Some of the rules that are key to our theoretical evaluationare presented. These include traditional reasoning rules such asSymmetry and Transitivity as well as variants of combination rulesexpressing monotonic reasoning, which have proven to be particularlyimportant and conclusive for the analysis of XML retrieval models.Furthermore, containment rules express important characteristics ofXML documents. By comparing the kind of rules a particular modelincorporates and the way it does so, we are able to provide an analysisof the behaviour of XML retrieval models.A further investigation of aboutness boundaries for particular retrievalmodels, which we call reflection, is the third step of the theoreticalevaluation. The final step in our theoretical evaluation isthe comparison with the pure type XML retrieval model to analysethe impact of structure on retrieval performance. This forth stepis an addition to the existing aboutness-based theoretical evaluationframeworks.In Chapter 5, we apply our theoretical evaluation methodologyto five successful XML retrieval models. We present an XML vectorspace model, two XML language modelling models and two furtherones, which have been specifically designed for XML retrieval. Foreach of these models, we go through all of the theoretical evaluationmethodology steps from Chapter 4 and draw conclusions on therepresentation of information in the model as well as its reasoningbehaviour. We are particularly interested in discussing how standardinformation retrieval models such as vector space retrieval have beenchanged to meet the requirements of XML retrieval and what kindof assumptions have guided the development of new models for XMLretrieval.Chapter 6 analyses specifics of the general XML retrieval evaluationframework. We formulate a theoretical evaluation of the experiChapter 1. Introduction 5mental evaluation for XML retrieval and point to potential inconsistenciesof the experimental evaluation dimensions for XML retrieval.We continue with our theoretical evaluation work specific to XMLretrieval, when also in Chapter 6 we analyse XML retrieval filteringused to deliver only answers that are most specific to an informationneed. We introduce a new theoretical methodology to analysefilters as aboutness decisions, before applying it to an XML retrievalfiltering model.Finally, Chapter 7 draws upon all the results from the previouschapters to demonstrate how theoretical evaluation insights can beused to explain results from mainstream experimental evaluations. Itrelates our theoretical evaluation results with the experimental onesfor XML retrieval to find out how the adjustment of existing flat documentretrieval models compares to the creation of completely newones, especially designed to meet the requirements of XML retrieval.For each of the XML retrieval experimental evaluation tasks, we shalldetermine the reasoning properties that support a good performanceand discuss the experimental performance of the XML retrieval modelsfrom Chapter 5. To our knowledge, no existing aboutness approachhas actually delivered such an in-depth analysis of experimentalresults using the insights from the theoretical evaluation.Our conclusion summarises the results and contributions of thework and considers critically advantages and limitations of our approachas well as possibilities for future work.