Cross Language Information Retrieval using Selective Documents Technique and Query Expansion

Author Name(s): Dr. Avinash J. Agrawal, A. V. Zadgaonkar
Author Email:


This paper is devoted to a new method that uses combined technique of pseudo relevance feedback and co-occurring term technique for query expansion to improve cross language information retrieval along with selective document technique to reduce search space. The base is an Information Retrieval (IR) system which uses cosine similarity technique to retrieve and rank documents and a multilingual module using dictionary based translation where the tf-idf values are calculated and stored in inverted index which is term based index. The threshold value is used as a selection medium for documents which reduces the search space of index reducing the overall time for retrieval. A Query Expansion (QE) module is added in to the said system. The aim is to use QE to overcome the limitations of dictionary based translation, and to retrieve more relevant results. The system is validated with several measures, which shows the difference in the result before expansion and after expansion.

Keywords:  Cross Language Information Retrieval, Cosine Similarity, Tf-idf, Pseudo Relevance Feedback, Query Expansion, HindiEnglish Dictionary



Cross Language Information Retrieval (CLIR) has been an important research field with the aim to provide users documents in languages different from that of query. A major challenge in CLIR is to bridge the language gap between documents and query. Query translation is now serving as a major cross-lingual mechanism in most current CLIR systems. A common approach in CLIR is to translate queries using dictionaries because of the simplicity and the availability of machine readable bilingual dictionaries. A major problem in dictionary-based CLIR systems is ambiguity. However, queries from users are often not clear or are too short, which produce more ambiguity in query translation, and reduce the accuracy of the cross language retrieval results. The language mismatch problem in CLIR is more serious than in monolingual information retrieval. Given a query in the source language, the translated query in the target language is built by selecting the “correct” translations from a list of candidate translations for each term in the initial query to build a structured query in the target language. There are two mutually exclusive techniques to address this problem. The single selection technique that tries to find one best translation for each term and the multiple selection technique, on the other hand, provides every possible translation available [8]. The detailed explanation about tf-idf, cosine similarity and inverted index is explained in previous work [5]. In this paper, pseudo relevance feedback technique is applied to improve the structured query translation by recalculating weights for query terms from top documents returned by the initial retrieval [6]. Then related terms are extracted from the top relevant documents using co-occurrence analysis, assuming that if two terms co-occur then they tend to be related. Our experiment is using a dictionary-based Hindi-English CLIR system which shows that this method helps to improve M.A.P. score. Experiments show that query expansion contributes only a minor improvement in precision; however it helps to retrieve more relevant documents. The combination of using query terms re-weighting and query expansion together is shown as a good solution.



We have retrieved the documents for the user query and shown the relevant documents. The tf-idf values were used for representing the documents and query in vector form and cosine similarity was computed to retrieve and rank the documents relevant to the query. The terms were stored in a inverted index were the corresponding documents in which the terms appears and their tf-idf values in descending order were also stored A threshold value was decided and documents with tf-idf greater than or equal to that value where only considered for computing cosine similarity. This helped in reducing the time complexity. Domain specific dictionary was built, the problem of ambiguity which arises when Hindi terms are converted in English is handled by considering all the different meaning of term in English for a particular Hindi term (multiple selection technique).Further we combined the techniques of pseudo relevance feedback and co-occurring term .where the weights of term in query were recalculated and then were used to expand the query and retrieve new set of documents. The experimental results show that the MAP score show improvement in the system after expansion as compared to system before expansion. The Prcision and Recall also shows improvement.

320 total views, 2 views today

Download PDF File

About the author: admin