The MapReduce model consists of two primitive functions: Map() and Reduce(). The main idea of this model is to hide details of parallel execution so as to allow users to focus only on their data processing tasks. Hadoop MapReduce is an open-sourced framework that supports MapReduce programming model introduced by Google. Finally, Section 6 concludes our discussion by summarizing our contributions.

Next, we elaborate on query processing using the proposed bitmap indexes (ColBit), which are then evaluated by a microbenchmark test. In turn, we propose a novel idea of columnar layout exploiting bitmap indexes. The following section discusses a rich body of existing literature related to our work.

The rest of this article is organized as follows. Finally, our join technique equipped with the bitmap indexes allows us to use only bitvectors for joining relations. Also, the bitmap indexes help save a large amount of I/O with compressed bitvectors. Moreover, bitmap indexes facilitate the processing of not only selection predicates but also star-join queries, which are widely used in data warehouses, on Hadoop. In addition, a modification of the Hadoop internals is not necessary in our approach. Both the layout transformation and the index-building tasks are performed at the loading time so that no overhead is imposed while processing data analysis tasks. Therefore, users do not need to know details about internal data layouts.

At loading time, ColBit automatically transforms data block replicas into their corresponding columnar layouts. It is achieved by both (i) skipping unnecessary data reads during analysis and (ii) reducing the size of overall intermediate data size through the substitution of most of intermediate results with compressed bitvectors. ColBit dramatically improves the performance of Hadoop-based programs by reducing a huge amount of I/O while processing data analysis tasks. In this regard, we propose ColBit, a combination of bitmap indexes and an efficient columnar data layout for data blocks stored on the Hadoop distributed file system, aka, HDFS. It is of critical importance to eliminate I/O bottleneck in Hadoop-based programs, considering a wide use of Hadoop in many scientific areas. Specifically, we focus on improving the I/O efficiency in massive data analysis when using Apache Hadoop. In the same line, this manuscript addresses the issue of I/O inefficiency in MapReduce-based programs. The community thus has exerted to obtain efficient I/O, especially by building new frameworks over Hadoop. Anderson and Tucek also noted that Hadoop was remarkably scalable but achieved very low efficiency per node, less than 5вЂ‰MB per second processing rate. Pavlo addressed that Hadoop MapReduce was 2 to 50 times slower than conventional parallel database systems except in the case of data loading. That said, MapReduce has raised a nontrivial concern of exhibiting a clear tradeoff between I/O efficiency and fault tolerance. Many algorithms for data analysis and mining have been rewritten for being run on Hadoop. Therefore, it has been widely recognized as an efficient tool for large-scale data analysis in the era of big data. Apache Hadoop, an open-sourced implementation of GoogleвЂ™s MapReduce, is a prominent data processing tool that processes a massive volume of data with a shared-nothing architecture in a parallel and efficient manner. We witness that the rate at which the data is being generated is faster, and the amount of the generated data is enormously larger than ever before. IntroductionДata volumes in scientific areas are unprecedently skyrocketing, and new sources and types of information are proliferating. We also provide Hadoop with indexing capability to save a huge amount of I/O while processing not only selection predicates but also star-join queries that are often used in many analysis tasks. We first incorporate a columnar data layout into the conventional Hadoop framework, without any modification of the Hadoop internals. In this article, we address the problem of the I/O inefficiency in Hadoop-based massive data analysis by introducing our efficient modification of Hadoop. While practitioners have rewritten many conventional analysis algorithms to make them customized to Hadoop, the issue of inefficient I/O in Hadoop-based programs has been repeatedly reported in the literature. Apache Hadoop has been a popular parallel processing tool in the era of big data.

Data analysis programs hive doot