Closed-Domain Question Answering System

Design, implementation and testing of a question answering system in the form of a chatbot

For the final project in the course ‘Conversational Agents’, I built a question answering (QA) system in a team with three other students. The QA-system takes questions about the Eiffel Tower as input and returns suitable answers. Answers are stored in a tree structure and were manually mapped to summarizing keywords, which form the nodes of the tree structure. Design choices were informed by previous research during the course with the Virtual Human Toolkit, which makes use of a simple probabilistic question answer matching technique. Including normalization techniques as well as natural language processing methods, we built a more sophisticated question answering system. A user study was conducted to evaluate the contribution of different components of the program concerning their influence on answer matching. The new system performed 10 times better in  correctly matching answers than the basic matching technique implemented in the Virtual Human Toolkit.

Example 1:
U7, Q63: “Where is the Eiffel tower build?”
U7, Q63-answer: “The Eiffel Tower is located in Paris.”

The system consists of several components, which are the graphical user interface, a normalization component, one component for advanced language processing and finally an answer unit. During normalization, non-standard ASCII symbols and punctuation are removed, all words are translated to lowercase, abbreviations are replaced by complete words and numbers are converted to words. The system contains a spell checker, based on Levenshtein distance. Finally, morphological analysis is done.

In a second step, several keywords are identified. This procedure makes use of part-of-speech tagging, identifying the respective word class. Nouns, verbs and adjectives were considered as containing the most relevant information in this closed-domain setting. Additionally, the interrogative pronouns “who”, “whom”, “where” and “why” are considered very important. A keyword list is created by looping through all words in the tagged question and adding the ones that meet the above criteria. The “w”- question words are first in the list (position 0), then nouns are inserted at position 1, adjectives at position 2 and finally verbs are appended. Thereby, the initial word order is destroyed. Several tests during implementation showed that ordering the words in this manner improves performance in comparison to adding the words to the list in the order that they occur in the initial question. Topic Identification is done by measuring the similarity of all keywords (and their synonyms) and the nodes in the answer tree (and their synonyms) using WordNet. Thereby, a main and a subtopic of a question are determined.

Example 2:
U7, Q63: extracted keywords [’where’, ’build’, ’eiffeltower’]
U7, Q63: mapped to main topic “location” and subtopic “build”

In the answer-unit, the answer sentence corresponding to the selected answer nodes is returned to the user via the GUI. The GUI and most of the normalization steps were implemented in C++, while the more advanced language processing is done with the help of the Python NLTK.