A Comparative Study of Open-Domain and Specific-Domain Word Sense Disambiguation Based on Quranic Information Retrieval

. Information retrieval is the process of analysing typed query as well as to retrieve relevant document according to the user query. Several issues can significantly affect the effectiveness of information retrieval. One of the common issue is the ambiguity lies on the words where a single word could yield several meanings. The process of identifying the exact sense of word is called Word Sense Disambiguation (WSD). Quran is the holly book for nearly 1.5 billion Muslims around the world. In particularly, Quran contains numerous words that can undergone multiple meanings. Therefore, there is a vital demand to apply WSD approach on Quran, in order, to improve the information retrieval. Several WSD approaches have been proposed for Quranic retrieval. However, these approaches are divided into two main categories; open-domain WSD approach and specific-domain WSD approach. Open-domain WSD is an approach that utilizes an open-domain dictionary such as WordNet, that is exploited to provide the exact sense. Whereas, domain-specific WSD approach aims to utilize a restricted training data that contain specific senses related to the domain of Quran. Hence, this study aims to establish a comparative study to investigate the two WSD categories including domain-specific and open-domain. For the domain-specific approach, a predefined example data has been collected to train Yarwosky algorithm which is a semi-supervised machine learning technique. Then, based on the training, such algorithm can classify the exact sense for the words. In contrast, WordNet which is an open-domain dictionary has been used in this study with semantic distances, in order, to identify the similarity between the query word and the results o f WordNet’s concepts. That dataset that has been used in this study is a Quranic translation. The experimental results have shown the mixed superiority of Yarwosky algorithm and WordNet WSD approach.


Introduction
Word sense disambiguation (WSD) is the task of identifying the correct meaning of a word in context.In any language, there are many words that yield multiple meaning for example, the word 'address' has two meanings; first it refers to a geographic location, and second it refers to examining something.Hence, WSD has been introduced to resolve such problem by identifying the exact sense of word [1].
As a basic semantic understanding task at the lexical level, WSD is a fundamental problem in natural language processing.It can be potentially used as a component in many applications, such as machine translation (MT) and information retrieval (IR).Recently, the application of WSD to MT, research has shown that integrating WSD in appropriate ways significantly improves the performance of MT systems [2].In the application to IR, WSD can bring two kinds of benefits.First, queries may contain ambiguous words (terms), which have multiple meanings [3].The ambiguities of these query words can hurt retrieval precision.Identifying the correct meaning of the ambiguous words in both queries and documents can help in improving the retrieval precision.Second, query words may have tightly related meanings with other words not in the query [4].Making use of these relations between words can improve retrieval recall.Overall, IR systems can potentially benefit from the correct meanings of words provided by WSD systems.
However, in the literature on the usage of WSD in IR, different researchers arrived at conflicting observations and conclusions.Some of the early research showed a dropped in retrieval performance by using word senses [5].Some other experiments observed improvements by integrating word senses in IR systems [6].This can be performed by identifying the exact sense for each word in the query which leads to increase the recall of retrieval [7].
There are less studies have been done on applying WSD on Quranic IR.This is probably due to the less of resources or the known mixed of results if WSD applied on IR.The recent studies of WSD on Quranic IR can be seen in Tiun et al [12] and Mohammad and Tiun [8].An issue risen by Tiun et al [12], is that they concluded that the failed WSD on domain-specific is due to the use of external knowledge like WordNet or LDOC and supervised learning technique is used.Thus, this encouraged them to use minimal supervised WSD in applying WSD on Quranic IR [12] [8], in which, the WSD is localised into the content of the information that need to be retrieved.
Nevertheless, it come to our conclusion that by applying WSD to IR, the IR accuracy should have some improvement since some believe that deep semantic analysis on the query process will give good impact on the IR performance, theoretically, and that could be the reason why WSD has been continuously studied.However, it is also wise to compare how much is the impact and how the results will behave when open-domain and closed domain of WSD studied on Quranic IR.Hence, this study will address this issue.

Related Works
Recently, researchers have focused on incorporating WSD with IR using various techniques.For example, Zhong and Ng [13] have proposed a novel language modeling approach to enhance information retrieval.Basically, the authors have used a unigram approach with negative Kullback-Leibler (KL) divergence as similarity distance function in order to link each document with its similar query.Then a supervised word sense disambiguation method has been used in order to expand the query words with their corresponding senses.In fact, the authors have used an open-domain source knowledge to train the WSD method on it using Chinese Treebank.The experiments have been performed on TREC dataset collection.As a result, the authors have reported a significant improvement when using the WSD with IR where the confidence level of retrieval was 95%.
Stevenson et al. [15] have proposed a domain-specific WSD approach for retrieving biomedical documents using the National Library of Medicine WSD dataset.Basically, the authors have used specific information related to the topic of each document where the ambiguous words are frequently occurred.This has been performed by implementing multiple approaches to extract lists of key terms that related to the medical subject heading terms.Such key terms have been used to represent the document topic in a knowledgebased WSD system.The authors have reported a significant enhancement when using such WSD approach by achieving a 70% of f-measure.
Zouaghi et al. [7] have proposed an open-domain WSD approach where WordNet lexicon has been combined with Lesk algorithm for Arabic information retrieval.In fact, the authors have manually collected Arabic documents.In addition, they have used Part-Of-Speech Tagging in order to tag the targeted words.Then the nouns have been migrated to the WordNet lexicon and then the Lesk algorithm has been used to identify the exact sense of words.Similarly, the authors have reported an enhancement when using WSD for Arabic IR by achieving 73% of f-measure.
Finally, Mohamed and Tiun [8] have proposed a domain-specific WSD approach for Quran information retrieval using a semi-supervised Yarwosky algorithm as classification method.In fact, a domain-specific data has been collected for the purpose of training.This data consists of two groups of Quranic verses; first verses with sense A, and second group contains verses with sense B. Hence, Yarwosky algorithm has been trained on such data in order to classify a testing data.After that, an information retrieval system has been built based on Yarwosky algorithm to retrieved relevant verses.The proposed method has improved the information retrieval by obtaining 77% of F-measure.

Research Method
This phase aims to accomplish the research objectives by applying two WSD approaches including Yarowsky algorithm (i.e.specific domain) and Lesk algorithm with WordNet (i.e.open-domain) and then establish a comparative analysis.In order to achieve such objectives, the research design will be illustrated as shown in Figure 1.In fact, the research design consists of five main phases: The first phase is associated with the dataset that has been used in this study by identifying its source, details and containing instances.The second phase is associated with the normalization tasks which aims to turning the data into a suitable format by removing unwanted data such as stop-words.The third phase is associated with the query modification.Such process aims to expand the query with corresponding synonyms or matching in order to identify the word's sense.This process has been applied twice; first using an open-domain dictionary (i.e.WordNet), and second using Yarowsky algorithm which will be trained on a specific-domain example set.The fourth phase aims to apply information retrieval based on the resulted modified query from both of WordNet and Yarowsky algorithm.Finally, the fifth phase aims to evaluate the two approaches by establishing a comparison between the two approaches WordNet and Yarowsky algorithm.

A Domain-Specific WSD (Yarowsky Algorithm)
Yarowsky algorithm is one of the machine learning technique that aims to classify the words' senses based on a relatively small set of training dataset [7].Such small set makes Yarowsky algorithm to be categorized as a semi-supervised machine learning technique.In fact, this algorithm is mainly depended on created predefined examples, where the verses will be divided into two groups of senses.First group contains the verses that are related to sense 1, whereas second group consists of the verses that are related to sense 2.More explanation on how the Yarowsky algorithm is carried out in the phase can be obtained from Mohamed and Tiun [8].

Query Modification
Query modification or so-called query expansion is the process of identifying corresponding synonyms, hyponymy or any semantic relatedness matching instances of the query's words in order to solve the ambiguity of the query [9].In fact, this phase is crucial because the process of word sense disambiguation is carrying out on it.For this purpose, WordNet dictionary has been used.Following sub-sections will illustrate the mechanism of modifying the query.

A WordNet Dictionary
This phase aims to identify the corresponding synonyms of the query/word.Such tasks would facilitate the process of identifying the exact sense for the query/word.For this manner, an open-domain dictionary which is WordNet has been used to provide the semantic correspondences.Table 1 shows the sample results of using WordNet.As shown in Table 1, the results of WordNet provide a definition for each concept in the query.Usually, WordNet bring various definitions that are related to multiple domains.For example, retrieving the definition of the word 'mouse' will be as 'quite person', 'animal' and 'electronic device'.In order to identify the exact definition, Lesk algorithm will be applied.Such algorithm can be described in the following section:

Lesk Algorithm
This algorithm is used to provide assumption that words in a given "neighborhood" tend to share a common topic [11].Lesk algorithm is usually used for identifying the most appropriate sense of definitions that brought from dictionary [12].For example, the word 'pray' has two senses located in the WordNet that can be shown in Table 2. Whereas, the word 'worship' has three senses that can be seen in Table 3.Each of the two senses of the word 'pray' is compared with each of the three senses of the word 'worship' and, it is found that the words 'deity' and 'worship' occurs in one sense each of the two words.These two senses are illustrated as follows: (i) Pray (address a deity, a prophet, a saint or an object of worship; say a prayer) "pray to the Lord".
These senses will be declared to be the most appropriate senses when the words 'pray' and 'worship' are used together.

Result and Evaluation
Since the evaluation will be based on retrieval effectiveness thus, the metrics that have been used is precision, recall and f-measure.Precision refers to the correctness of retrieval answers, while recall refers to the correctness of not retrieved answers.Such metrics can be calculated as the following Equations (1 and 2): where TP is the number of correct retrieved answers, FP is the number of incorrect retrieved answers, and FN is the number of correct answers but not retrieved.Hence, it is possible to calculate the overall accuracy which called f-measure based on the following Equation ( 3

Evaluation of the 'day' verses
This section will address the results of the two WSD approaches when using the 'day' verses.The evaluation has been performed based on the effectiveness of retrieval.Table 4 shows such results.Table 4. Results of the two WSD approaches using the word 'day'.

Evaluation of the 'charity' verses
This section will address the results of the two WSD approaches when using the 'charity' verses.The evaluation has been performed based on the effectiveness of retrieval.Table 5 shows such results.
Table 5. Results of the two WSD approaches using the word 'charity'.

Evaluation of the ''charity' verses
This section will address the results of the two WSD approaches when using the 'pray' verses.The evaluation has been performed based on the effectiveness of retrieval.

Evaluation of the 'light' verses
This section will address the results of the two WSD approaches when using the 'light' verses.The evaluation has been performed based on the effectiveness of retrieval.Table 7 shows such results.
Table 7. Results of the two WSD approaches using the word 'light'.

Comparison between the Four Words
In this subsection, a comparison has been taking a place in order to compare each mentioned word using the two WSD approaches Yarowsky and WordNet.The evaluation has been performed based on F-measure and the average for the two senses.Table 8 shows the results for each word.As shown in Figure 2, Yarowsky WSD approach has outperformed the WordNet approach for two words including 'charity' and 'pray' by obtaining 77% and 39% of Fmeasure respectively.This is because these words contain specific senses that are related to the domain of Quran.Since the Yarowsky has been trained on a domain-specific data thus, the superior of Yarowsky results is expected.In contrast, WordNet WSD approach has outperformed Yarowsky for the word 'light' by achieving 64% of F-measure.This is because such word contains senses that tend to be more domain-less.Finally, the classification of the sense word 'day' has shown a similar performance for both Yarowky and Lesk, with WordNet by achieving a 65% of F-measure.This is because such word has two senses; the first is domain-specific which is 'judgment day' (i.e.related to Quran), and the second is domain-less which is 'day and night'.

Discussion and Conclusion
The experimental results have shown a superiority for Yarowsky WSD approach in terms of retrieving words that have specific sense in the Quran, meanwhile do not have specific sense in the WordNet.For instance, the word 'day' has several senses in WordNet but only one sense could be occurred in Quran which is 'day and night', however in Quran the word day has been associated with 'judgment day' such sense is not located in the WordNet.In contrast, WordNet approach has shown a superiority in terms of retrieving words that have been used in Quran generally which indeed WordNet dictionary will identify its sense.For instance, the word 'light' has many senses in WordNet in compared to the Yarowsky which trained on two senses only.That is why WordNet is able to achieve higher results for the word 'light'.This emphasizes that the domain-specific WSD approach has a significant impact on improving the retrieval using restricted domains such as Quran.To sum up, the research finding of this study conclude the mixed of results of open-domain WSD when applied on Quranic IR.Therefore, probably a careful study of designing WSD technique should be applied on domain-specific IR system, in a way the mixed of both resources; external knowledge like WordNet and localised semi-supervised can be considered in order to have the best of both approaches.
prayS: (v) pray (address a deity, a prophet, a saint or an object of worship; say a prayer) "pray to the Lord"

Table 2 .
Senses of the word 'pray'.

Table 3 .
Senses of the word 'worship'

Table 6
shows such results.

Table 6 .
Results of the two WSD approaches using the word 'pray'.

Table 8 .
Comparison between all the words Results of the comparison between all the words.