Author Email: firstname.lastname@example.org
Most organizations today store their data in a variety of data storage systems such as relational databases, columnar stores, document stores, text search engines, and so forth. The adoption of multiple data storage systems tailored towards specific needs has its own challenges. The developers using these specialized systems have to often integrate several of them together. In the absence of a uniform framework, multiple engineers have to independently develop optimization logic and support different data retrieval mechanism for each system. The framework proposed in this paper intends to solve these problems by providing a uniform query framework to retrieve data from heterogenous systems. This paper also analyzes the response time of query framework with respect to individual data sources.
Keywords Data Management, Big Data, Heterogenous Data Source, Information Retrieval, NoSQL, Query Framework, RDBMS
For over four decades relational database management systems have dominated the industry of data storage systems. However, there is a lot of interest and adoption of other of other types of data storage systems over the last decade such as document stores, columnar stores, stream processing engines, text search engines etc. In 2005, Stonebraker and Çetintemel  argued that these new types of data storage systems, also known as NoSQL databases, would bring an end to the “one size fits all” paradigm and can offer cost effective performance. Their argument today seems more relevant than ever as many specialized open-source data systems have since become popular such as Cassandra  (columnar store), MongoDB  (document store), Elasticsearch  (text search), etc. As organizations have invested in data processing systems tailored towards their specific needs, they are faced with some challenges. The developers using these specialized systems have to often integrate several of them together. In the absence of a uniform framework, multiple developers have to independently develop optimization logic and support different data retrieval mechanism for each system.
Advance data management systems and associated practices continue to evolve resulting in organizations embracing multiple heterogenous database systems. At the same time, relational data sources, accessed through SQL, remain an essential means to how organizations work with the data. In such a scenario, a uniform framework provided in this paper can play an important role with its support for traditional relational databases as well as NoSQL database systems. It should also be noted that some of the operations such as text search, joins, aggregates which are not natively supported on NoSQL databases can be performed through this framework. For instance, the proposed framework allows filtering on Cassandra tables although Cassandra does not have native support for it. Similarly, the framework allows joins operations on MongoDB collections although MongoDB does not have native support for such operations. The results shown above suggest that the time taken by the framework is more than the time taken by the query if executed on the data source itself. This increased time can be attributed to the time taken by the framework for query parsing, validation and optimization. There is scope for further analysis and performance tuning of the framework. On the other hand, the concept of having a uniform query framework for multiple heterogenous data sources and the ability to perform operations such as inter-database joins, aggregates, filters and nested queries is extremely useful and can change how organizations integrate various database systems
320 total views, 1 views today