Overview and Application of Text Data Pre-Processing Techniques for Text Mining on Health News Tweets

Author Name(s): Gauri Chaudhary, Dr. Manali Kshirsagar
Author Email: chaudhary_gauri@yahoo.com

Abstract

With the huge abundance of textual data in electronic form on the internet, there is a growing need to identify and extract meaningful information and patterns from this underlying text data. While in a broader context this is Data Mining, since applied to text data, this gives rise to a specialization within data mining, called text mining. Various useful applications of text mining are clustering, classification, trends analysis etc. This paper focuses on various techniques of pre-processing the text data so that it is transformed into a form to which text mining can be applied. Various data preprocessing techniques have been applied to health news from more than 15 major health news agencies collected using Twitter API, available on UCI Machine Learning Repository and output at the end of each pre-processing step is specified.

Introduction

In last few decades there has been vast increase in textual data in electronic form such as collection of web pages, research papers, emails etc. Previously data mining was meant for structured data such as that stored in relational databases or data warehouses. Today, all information available with various organizations is stored electronically in text databases (or document databases). Text data is semi-structured because it is neither structured nor unstructured. Free flowing text stored on the web is an example of unstructured data. In order to extract useful information i.e. interesting patterns from unstructured textual data, we use text mining techniques. Text mining is analysis of text data and extraction of knowledge from this data. In order to analyze and perform any kind of operation on text data, the text data needs to be transformed to numbers. Only then can any specific text mining technique be applied to it. [05]. There is a need to extract data from large number of documents, compare them, find similar documents, select relevant information from them, and find patterns across multiple documents. Hence text mining is applied on data stored in large document collections. Text mining involves various tasks like Information Retrieval, Classification, Clustering, Summarization, Trends Analysis, Association Analysis, Visualization etc. The objective of this research is preparing or preprocessing data for text mining. Fig. 1.0 below describes the complete system flow for text mining. It starts with collection of textual data (also called the document corpus) from one or more sources (example: web pages or emails or digital libraries).

Conclusion

Text mining has been increasingly important in recent times due to the huge creation and usage of electronic documents. The objective of text mining is to extract useful information such as patterns, trends or knowledge from textual data. It is essential to convert the text data to numbers to perform text mining. Various pre-processing methods are described in this paper to convert the text documents to a form suitable for further knowledge mining. The text preprocessing techniques such as tokenization, stop words removal, stemming and document representation as TF-IDF vectors have been described and their application demonstrated on Health news tweets dataset. Thus the researchers should be able to understand and apply the pre-processing techniques described in this paper for any text mining applications.

997 total views, no views today

Download PDF File

About the author: admin