Hadoop Notes


BIG DATA

Big Data is a collection of data sets so large(beyond storage of single machine) and complex(unstructured and semi structured). It is difficult to process such a huge and complex data by using traditional database management tools and data processing applications.

Big data brings with it two fundamental challenges:
1) How to store and work with voluminous data sizes
2) How to understand data and turn it into a competitive advantage.

The volume of data is growing exponentially . Earlier we used to talk about data in terms of MB’S, GB’S.  But the time is arrived when we talk about data volume in terms of TB’S ,PB’S ,ZB’S. Global data volume was around 1.8 ZB’S in 2011 and is expected to be 7.9Z.B’S in 2015. It is also known that the global information double in every two years.

How analysis of Big Data is useful for organization?
Effective analysis of big data provides a lot of advantages . organizations easily  learn which areas more  important  to focus on , which areas less important . big data analysis provides some early key instances that can prevent the company form a huge loss (or) helps in grasping a great opportunity with open hands. A precise analysis of big data helps in decision making . for instance now a days people rely so much on face book  and twitter before purchasing any product or service.

What is hadoop ?
Hadoop is open source framework , distributed ,batch processing and fault –tolerance system which is capable of storing huge amount of data[TB, PB,ZB], and it is also capable of creating distributed applications that process huge amount of data.

History of hadoop:
In the year 2004 Google release a paper called GFS (Google file system) and MapReduce.

GFS:-  It is a distributed frame work. It is use to store and manage huge amount of data in a cluster of machine.
Google follows some proprietary standards for arrangement of data i.e., GFS can understand only Google file system it is not a Generic file system.

Map Reduce :- It is programming frame work it is used  to process data present in GFS.

In the year 2004, apache started implementation of this paper.
Hadoop was implemented by  Doug Cutting  and Mike Cafferella (they  are team leads in apache Nutch team).
Hadoop was implemented in the  year 2005. Upto 2007 it was in Alpha stage (testing stage). The first beta version of hadoop (Hadoop 0.14.1) is released in the year 2007 Nov 4th .
   The versions are :- Hadoop – 0.14.2 .......haoop- 0.15.0.....hadoop-0.20.0......hadoop-1.0.4 ,hadoop- 1.1.2.....
             ..............hadoop-2.0.0,hadoop-2.1.1,hadoop-2.2.0


Note:
Hadoop was first conceived to fix a scalability issue that existed in Nutch( open source crawler and search engine) in the year 2005.
upto 2007 it was in testing. The first version of hadoop(hadoop-0.14.0) is released in the year 2007 nov 4th.

Hadoop framework consists two main core components .

HDFS:
It is a distributed file system.
It is used to store and manage huge amount of data in cluster of machines. It is modelled after Google’s paper on GFS. It is the enhancement of hdfs.
Hdfs store large amount of information
It is scalable ,provides fast access to this information
Fault Tolerance:  by detecting faults and applying quick , automatic recovery .
Processing logic close to the data rather than the data close to the processing logic.
Portability across heterogeneous commodity hardware and O/S.
Efficiency : by distributing data and logic to process it in parallel on nodes where data is located.
Reliability by automatically maintaining multiple copies of data and automatically redeploying              ,processing logic in the  event of failure.

Features of HDFS:
Support for very large files
Commodity H/W: hdfs requires a commodity H/W[ the h/w which is been available for most of the vendors] and hadoop does not require high configuration hardware, expensive S/W to be part of it’s basic installation . for the commodity h/w chance of  node failure across the cluster is high at least for large clusters.

Streaming data access: hdfs is the most efficient data processing pattern. That pattern is write once read many times.  It supports streaming data access[sequential flow, no random index access]

Once we can store a file on top of hdfs file ,we can’t override  that file ,instant we can take the file into local environment. We modify the content and we can put the samefile on top of the hdfs file. By deleting whole hdfs file.

High latency data access: generally applications that require low latency access data.

As we are using in hdfs ,used to very large amount of data . because  it takes more amount of time . this may be at the expense of latency

MAPREDUCE:
It is batch- based distributed programming frame work modeled after Google’s paper on MapReduce. It  is the enhancement of Google mapreduce. It  is used to process huge amount of datasets present in hdfs in parallel.
It allows you to parallelize work over a large amount of raw data, such as combining web logs with relational data.
The MapReduce model simplifies parallel processing by abstracting away the complexities involved in working with distributed systems, such as computational parallelization,
work distribution, and dealing with unreliable hardware and software.
To implement distributed  applications using mapreduce applications we need a language.
     1)java 2)python 3)scala 4)c#.net

When we start hadoop  following  process will be started:-
i) Name Node daemon(Thread)
ii) Data Node
iii) Secondary Name Node
iv) Job tracker
v) Task tracker
The above daemons are running behind hadoop frame work.


In cluster we need to install following softwares.
Operating system in each machine in the cluster (suggested O/s is Linux(ubuntu linux, Cent os,       RedHat linux).
Java software must be installed in all machine.
Hadoop must be installed in all machines.
To implement Map reduce programs we need some editor ( ex: Eclipse or NetBeans).
Name node is always configure in maser machine.
Data node is must be configure in each and every slave.

Note:
HDFS follows master and salve architecture.In case of HDFS , Name node is the Master daemon and DataNode is the slave daemon.

Note:  To identify a particular system in the cluster, we have to assign one unique ip address for each machine in the cluster . By default , when we install OS systems , it comes  with default  IP Address (127.0.0.1),host name localhost .
It is difficult to remember ID  address . Since it contains 12-digits , so it is better to assign unique host name to each machine.

To change the host name of the machine ,open following file from the terminal with administrative permission.
In ubuntu linux:  /etc/hostname
         
         In  Master machine
       
   $sudo gedit  /etc/hostname

   change the name localhost to master.
       
         In Slave machine One:
       
   $sudo gedit  /etc/hostname

   change the name localhost to  slave1.
       
         In Slave machine Two:
       
   $sudo gedit  /etc/hostname

   change the name localhost to  slave2.
         
          To configure IP address with host name open following file from the terminal
             
              In master machine

              $ sudo gedit /etc/hosts
         
                Localhost      127.0.0.1 (default entry in hosts file)
             
           We have to change it to

master           192.168.0.1

           in slave one  machines

  $sudo gedit  /etc/hosts

                Localhost     127.0.0.1

  Slave1      192.168.0.2

Configuration files :
i) When we install hadoop it comes with some set of configuration files.
For ex:- core-site.xml
               Hdfs-site.xml
               Mapred-site.xml
                .........................
           
Configuration file is collection of properties
A property is collection is name and value pair.
To configure  NameNode in particular machine open ‘core-site.xml’  either with  Gedit window or VI editor and set following properties.

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>
</configuration>

fs.default.name   is a standard property which is used to configure NameNode.

Here master is a host name.
9000  is default port number for  the NameNode .
 if we want ,we can change port number.
                     When we copy the file into hadoop cluster from one of the slave then , copy request goes to NameNode by
                 the  HDFS client. In these case name node verifies two things:-
Whether client has write permission or not.
Whether the file is already existed or not  in the given path.

           If client does not have write permission or file is already existed then we will be notified with IOE exception. otherwise give file will be divided into smaller chunks called blocks. Default block size is 64MB. Block size cannot  be decrease ,but it can be increased. New block size should be multiples of 64.
Ex: 64MB,128MB,192MB,256MB........


To configure new block size open ‘hdfs-site.xml’ set of following properties
<configuration>
<property>
<name>dfs.blocksize</name>
<value>
64*1024*1024 //value should be in terms of bytes
Or
128*1024*1024
                </value>
</property>
</configuration>

File blocks are stored in DataNode machines only.
Location of each block is decided by NameNode.
For every 3sec data node sends ‘heartbeat message’ to the NameNode.

Heart Beat Message contains
New blocks which are added to machine.
Blocks which are deleted from the machine .
Blocks which are corrupted from the machine

If any data node is dead due to some hardware problems or O/S problems,  then there  is no heartbeat communication between NameNode and DataNode.

If there is no heartbeat upto 10mins, then NameNode decides with that node is  dead.

Name node maintains two types of list:-
A) Live Node list
B) Dead  Node list.

Live node list contains IP addresses/Host names of those machines which are running.
Dead node list contains IP address/ Host names of those machines which running.

Replication :
It is nothing but reputation i.e., hadoop frame work maintains 3 replicas of each block in a file by default, to provide more data guaranty. We can increase or decrease replication factor by setting following property in ‘hdfs-site.xml’;
<property>
<name>dfs.replication</name>
<value>3</name>
</property>
Note: When we submit map reduce program to cluster to process requested file, it will process only one replica of each block which is nearest to the client position. Internally hadoop framework uses on algorithm called ‘shortest distance path algorithm’ to find nearest replica towards each block from client position.

Name Node is maintain total file system metadata in the form of fsimage and editlog.

File system metadata nothing but number of files which are present in cluster, number of blocks towards each files, location of each block towards each file and so on.

Filesystem Metadata in HDFS

 How the NameNode persists filesystem metadata ?
HDFS metadata changes are persisted to the edit log.

NameNode’s primary responsibility is storing the HDFS namespace(like the directory tree, file permissions, and the mapping of files to block IDs).It’s important that this metadata (and all changes to it) are safely persisted to stable storage for fault tolerance.
This filesystem metadata is stored in two different constructs: the fsimage and the edit log. The fsimage is a file that represents snapshot of the filesystem’s metadata. It is not suitable for making small incremental updates like renaming a single file changing permission. Instead of  writing a new fsimage every time when  the namespace is modified, the NameNode records the modifying operation in the edit log for durability. If the NameNode crashes, it can restore its state by first loading the fsimage then replaying all the operations (also called edits or transactions) in the edit log to catch up to the most recent state of the namesystem.
The edit log comprises a series of files, called edit log segments, that together represent all the namesystem modifications made since the creation of the fsimage.

Why is Checkpointing Important?
A typical edit ranges from 10s to 100s of bytes, but over time enough edits can accumulate to become unwieldy. A couple of problems can arise from these large edit logs. In extreme cases, it can fill up all the available disk capacity on a node, a large edit log can substantially delay NameNode startup as the NameNode reapplies all the edits. This is where checkpointing comes in.
Checkpointing is a process that takes an fsimage and edit log and compacts them into a new fsimage. This way, instead of replaying edit log, the NameNode can load the final in-memory state directly from the fsimage. This is a far more efficient operation and reduces NameNode startup time.

Checkpointing creates a new fsimage from an old fsimage and edit log.
However, creating a new fsimage is an I/O- and CPU-intensive operation, sometimes taking minutes to perform. During a checkpoint, the namesystem also needs to restrict concurrent access from other users. So, rather than pausing the active NameNode to perform a checkpoint, HDFS defers it to either the SecondaryNameNode or Standby NameNode,
In either case though, checkpointing is triggered by one of two conditions: if enough time has elapsed since the last checkpoint (dfs.namenode.checkpoint.period), or if enough new edit log transactions have accumulated (dfs.namenode.checkpoint.txns). The checkpointing node periodically checks if either of these conditions are met (dfs.namenode.checkpoint.check.period), and if so, kicks off the checkpointing process.
Checkpointing with a SecondaryNameNode
The Secondary NameNode is one which constantly reads all the file systems and metadata from the RAM of the NameNode and writes it into the hard disk or the file system. It is responsible for combining the editlogs with fsimage from the NameNode. It downloads the EditLogs from the NameNode at regular intervals and applies to fsimage. The new fsimage is copied back to the NameNode, which is used whenever the Namenode is started the next time.
If the NameNode system fails, the entire Hadoop HDFS goes down and you will lose the entire meta data  .when we start new NameNode,it will get file metadata from the SNN. By default checkpoint is trigged for every one hour.


Note: In the HDFS Architecture, the name – Secondary NameNode  is not a substitute of the NameNode.

Functions of a NameNode:
Let’s list out various functions of a NameNode:
1. The NameNode maintains and executes the file system namespace. If there are any modifications in the file system namespace or in its properties, this is tracked by the NameNode.
2. It directs the Datanodes (Slave nodes) to execute the low-level I/O operations.
3. It keeps a record of how the files in HDFS are divided into blocks, in which nodes these blocks are stored and by and large the NameNode manages cluster configuration.
4. It maps a file name to a set of blocks and maps a block to the DataNodes where it is located.
5. It records the metadata of all the files stored in the cluster, e.g. the location, the size of the files, permissions, hierarchy, etc.
6. With the help of a transactional log, that is, the EditLog, the NameNode records each and every change that takes place to the file system metadata. For example, if a file is deleted in HDFS, the NameNode will immediately record this in the EditLog.
7. The NameNode is also responsible to take care of the replication factor of all the blocks. If there is a change in the replication factor of any of the blocks, the NameNode will record this in the EditLog.
8. NameNode regularly receives a Heartbeat and a Blockreport from all the DataNodes in the cluster to make sure that the datanodes are working properly. A Block Report contains a list of all blocks on a DataNode.
9. In case of a datanode failure, the Namenode chooses new datanodes for new replicas, balances disk usage and also manages the communication traffic to the datanodes.



DataNodes:
Datanodes are the slave nodes in HDFS, Unlike NameNode, datanode is a commodity hardware, that is, a non-expensive system which is not of high quality or high-availability. Datanode is a block server that stores the data in the local file.
Functions of DataNodes:
Let’s list out various functions of Datanodes:
1. Datanodes perform the low-level read and write requests from the file system’s clients.
2. They are also responsible for creating blocks, deleting blocks and replicating the same based on the decisions taken by the NameNode.
3. They regularly send a report on all the blocks present in the cluster to the NameNode.
4. Datanodes also enables pipelining of data.
5. They forward data to other specified DataNodes.
6. Datanodes send heartbeats to the NameNode once every 3 seconds, to report the overall health of         HDFS.
7. The DataNode stores each block of HDFS data in separate files in its local file system.
8. When the Datanodes gets started, they scan through its local file system, creates a list of all HDFS data blocks that relate to each of these local files and send a Blockreport to the NameNode.


MAPREDUCE:
It is batch- based distributed programming frame work modeled after Google’s paper on              MapReduce. It  is the enhancement of Google mapreduce. It  is used to process huge amount of datasets present in hdfs in parallel.
It allows you to parallelize work over a large amount of raw data, such as combining web logs with relational data.
The MapReduce model simplifies parallel processing by abstracting away the complexities involved in working with distributed systems, such as computational parallelization,
work distribution, and dealing with unreliable hardware and software.
To implement distributed  applications using mapreduce applications we need a language.
1)java 2)python 3)scala 4)c#.net

To configure Job Tracker in master machine, go to mapred-site.xml , set following property
<configure>
<property>
<name>mapred.job.tracker</name>
<value>master:9001</value>
</property>
</configuration>

Note : In case of small clusters JobTracker will be configured in the NameNode machine.In case of Big clusters JobTracker will be configured in a separate machine.

Task tracker will configure in each and every slave (data node).
Job tracker is responsible for receiving job from the Job queue which is having highest priority and assign to the TaskTracker.

Note: In case of real time production cluster we can except multiple job simultaneously all the submitted job are stored in a queue called job queue.

In order to schedule the job in a cluster, JobTracker uses different schedulers like
1) FICO scheduler( FIRST IN FIRST OUT)
2) Capacity scheduler
3) Fair scheduler
 In gen-1 hadoop (it is also called as classic mapreduce) default scheduler is FIFO. In case of FIFO we can schedule only one job at a time in the cluster(multiple jobs can’t be scheduled).

Note : In a job queue user requests are stored based on priority.

JobTracker receives the job from the job queue communicates NameNode to get ip address of those machines in which requested file data is present. After receiving ip address then JobTracker assign job to the TaskTrackers in which they are running. Task tracker sends heart beat to the job tracker for every 100 sec by default.
Heart beat contains  Job status information ,Counters information and so on.

Note: If any task tracker is failed during map reduce program execution map reduce job will not be halted. It will to be reschedule in another machine automatically.
If the job tracker is failed all the map reduce jobs which are running in the cluster will be halted.


This is the single point of failure in Gen-1 hadoop we can overcome this problem in
Gen-2(Yarn).

Job is collection of tasks.

A mapreduce  job is collection of following task:
1) Mapper task.
2) Combiner task.
3) Partitioner task.
4) Reducer task.

Note: Combiner and partitioner task both are optional.

Every task tracker comes with some set of slots by default(slots are nothing but JVM instances). By default task tracker come with 2 Mapper slots (2JVM instanes) and 2reducer slots(2JVM instance). We can increase or decrease mapper or reducer slots using following properties present in ‘mapred-site.xml’.
A) <property>
<name>mapred.tasktracker.map.tasks.maiximum</name>
<value>2(default) change to 1 o 3 or 5 upto 40 </value>
</property>

B) <property>
<name>mapred.task tracker.reducer.tasks.maximum</name>
<value>2(default) change to 1 or 3 or 5 upto 40</value>
</property>

Maximum number of mapper or reducer slots depending on system configuration (Ram speed, Processor speed, Mother board configuration). We can configure up to 40 mapper and 40 reducer slots.

Block is input to the mapper task i.e., for every block they will be one mapper task.
Mapper output stored in spil temporary in the form of key and value pair. At mapper side 3 spils are merged into single spil (at mapper side margining factor is 3 by default). After merging all the spils into single spil first data will be stored by key automatically by the framework which will be stored in local drive in the form of sequence file.

Every reducer slot is configured with 5 threads by default . they are used to retrieve mapper outputs which will be written  into reducer system. This process is called shuffling.
At reducer side default spil margining factor is 10. After margining all the mapper outputs into single spil,then the data will be sorted by key (this is called sorting phase). This sorted spil is the input to the reducer task.
Reducer also emits the output in the form of key, Value pairs which will be stored in HDFS .

Depends on the type of the file appropriate input formatter class creates input spilt for every block .

For ex: 1)In case of text file :- TextInputFormat predefined class create input split.
2)In case of xml files :- XmlInputFormat predefined class creates input split and so on.

Note :- Block belongs HDFS, where as input split belongs to map reduce which is create in RAM temporary.

Input split is a collection of records. Record is a collection of key and value. Key is the offset code which is generated by Hadoop frame work automatically starts with zero(0).
Input split is input to the mapper task. For every mapper task they should be 1 supper class called ‘Mapper’ . It takes 4 arguments .
1) input key
2) Input value
3) output key
4) output value.
Life cycle methods of mapper :-
1) setup() :- It is a first life cycle method it is called by hadoop frame work, At that time of loading mapper into the memory. It is executed only once (1 time). It is used to perform initialization operation (variable initialization, data base connections, file open and so on).            
2) Map method :- It is executed for every record present in the input split. It is used to perform basic operation. It takes off set code as a input key, value as a input value.
3) Cleanup method :-  It is also execute only once by the frame work. It is used to perform cleanup operations like file closing, data base closing, socket closing and so on.


The Parts of a Hadoop MapReduce Job
The user configures and submits a MapReduce job (or just job for short) to the framework, which will decompose the job into a set of map tasks, shuffles, a sort, and a set of reduce tasks.The framework will then manage the distribution and execution of the tasks, collect the output,and report the status to the user.
The job consists of the parts as shown below
   
Part Handled By

Configuration of the job User

Input splitting and distribution Hadoop framework

Start of the individual map tasks with their input split     Hadoop framework

Map function(it is called once for each input key/value pair)     User

Shuffling and sorting         Hadoop framework

Reduce function(it is called once for each key/value
emited by mapper                                                                user

Collection of the output and storage in the
configured job output directory in N parts,        Hadoop framework where N is the number of reduce tasks

The user is responsible for handling the job setup, specifying the input location(s), specifying the input, and ensuring the input is in the expected format and location. The framework is responsible for distributing the job among the TaskTracker nodes of the cluster; running the map, shuffle, sort, and reduce phases; placing the output in the output directory; and informing the user of the job-completion status.

For the framework to be able to distribute pieces of the job to multiple machines, it needs to fragment the input into individual pieces, which can in turn be provided as input to the individual distributed tasks. Each fragment of input is called an input split. How input splits are constructed from the actual input files depending on configuration parameters and the class that actually reads the input records. An input split will normally be a contiguous group of records from a single input file.


The framework will make one call to your map function for each record in your input. There will be multiple instances of your map function running,  in multiple Java JVMs), and potentially on multiple machines. The framework coordinates all of this for you.



Configuring a Job

All Hadoop jobs have a driver program that configures the actual MapReduce job and submits it to the Hadoop framework. This configuration is handled through the JobConf or Job object,which is used to submit a job to the Hadoop framework for execution.


 you need to set JobConfig or Job object with required parameters for
the job. These include the input and output directory locations, the format of the input and output, and the mapper and reducer classes.

All jobs will have a map phase, and the map phase is responsible for handling the job input. The configuration of the map phase requires you to specify the input locations and the class that will produce the key/value pairs from the input.


Element Required? Default

Input path(s)                                    Yes

Class to read and convert the input path
elements to key/value pairs              Yes
value pairs

Map output key class                             No         Job output key class

Map output value class        No       Job output value class

Class supplying the map function Yes

Suggested minimum number of map tasks No Cluster default

Number of threads to run in each map task No 1

Most Hadoop Core jobs have their input as some set of files, and these files are either a textual key/value pair per line or a Hadoop-specific binary file format that provides serialized key/value pairs.
The class that handles the key/value text input is KeyValueTextInputFormat.

The class that handles the Hadoop-specific binary file is SequenceFileInputFormat.
Specifying Input Formats



The Hadoop framework provides a large variety of input formats. The major distinctions are between textual input formats and binary input formats. The following are the available formats:
• KeyValueTextInputFormat: Key/value pairs, one per line.
• TextInputFormant: The key is the line number, and the value is the line.
• NLineInputFormat: Similar to KeyValueTextInputFormat, but the splits are based on N lines of input.
• MultiFileInputFormat: An abstract class that lets the user implement an input format that aggregates multiple files into one split.
• SequenceFIleInputFormat: The input file is a Hadoop sequence file, containing serialized key/value pairs.
KeyValueTextInputFormat and SequenceFileInputFormat are the most commonly used input formats.

Configuring the Reduce Phase
To configure the reduce phase, the user must supply the framework with five pieces of information:
• The number of reduce tasks; if zero, no reduce phase is run
• The class supplying the reduce method
• The input key and value types for the reduce task;

by default, the same as the reduce output
• The output key and value types for the reduce task
• The output file type for the reduce task output

Setting the number of reduce tasks and the reducer class.
The configured number of reduce tasks determines the number of output files for a job that will run the reduce phase. Tuning this value will have a significant impact on the overall performance of your job. The time spent sorting the keys for each output file is a function of the number of keys. In addition, the number of reduce tasks determines the maximum number
of reduce tasks that can be run in parallel.

The framework generally has a default number of reduce tasks configured. This value is set by the mapred.reduce.tasks parameter, which defaults to 1. This will result in a single output
file containing all of the output keys, in sorted order. There will be one reduce task, run on a single machine that processes every key.
The number of reduce tasks is commonly set in the configuration phase of a job.
             conf.setNumReduceTasks(1);

The method runJob() or waitForComplection() submits the configuration information to the framework and waits for the framework to finish running the job. The response is provided in the job object.
The RunningJob class provides a number of methods for examining the response. Perhaps the most useful is job.isSuccessful().

Once the job finishes, the framework will provide you with a filled-out RunningJob object. This object has information about job status via the
conf.isSuccessful() method. The framework will report that the job was unsuccessful if it was unable to complete any single map task or if the job was killed.
Note: The machines in the cluster are not required to be homogeneous, and commonly they are not. The cluster machines may even have different CPU architectures and operating systems. But if the machines have similar processing power, memory, and disk bandwidth, cluster administration is a lot easier, because in that case, only one set of configuration files and runtime environments needs to be maintained and distributed. A cluster will have one JobTracker server,one NameNode server, and one secondary NameNode server, and DataNodes and TaskTrackers. The JobTracker coordinates the activities of the TaskTrackers, and the NameNode manages the DataNodes.

In the context of Hadoop, a node/machine running the TaskTracker or DataNode server is considered a slave node. It is common to have nodes that run both the TaskTracker and DataNode servers. The Hadoop server processes on the slave nodes are controlled by their respective masters, the JobTracker and NameNode servers.

JobTracker: The JobTracker provides command and control for job management. It supplies the primary user interface to a MapReduce cluster. It also handles the distribution and management of tasks. There is one instance of this server running on a cluster. The machine running the JobTracker server is the MapReduce master.

TaskTracker: The TaskTracker provides execution services for the submitted jobs. Each TaskTracker manages the execution of tasks on an individual compute node in the MapReduce cluster. The JobTracker manages all of the TaskTracker processes. There is one instance of this server per compute node.

Balancer: During normal usage, the disk utilization on the DataNode machines may become uneven. This is particularly common if some DataNodes have less disk space. The Balancer moves data blocks between DataNodes . The Balancer will also rebalance the cluster as new DataNodes are added to an existing cluster. The Balancer is not a started automatically.It must be run by the user via the command
bin/hadoop balancer [-threshold<threshold>]. The optional argument is the maximum amount of variance in disk space utilization between DataNodes for the cluster to be considered balanced. The default is 10%. As of Hadoop 0.19.0, this is not a configuration parameter.These server processes are typically started once per cluster instance. DataNodes and TaskTrackers may be dynamically added and removed from a running cluster Hadoop Core configuration: The Hadoop Core is configured by two XML files:

hadoop-default.xml and hadoop-site.xml. hadoop-default.xml provides reasonable defaults and comes with the Hadoop distribution. The configuration provided by this default file is suitable for a single machine instance.



File Types for MapReduce Jobs
The Hadoop framework supports text files, binary (sequence) files, and map files, which are actually a pair of sequence files. Let’s take a closer look at each of these file types.
Text Files
The Hadoop framework supports a number of textual input files and output files. The input formats support transparent decompression of input files if an input file name ends in one of the recognized compression format suffixes (.gz, .deflate, .lzo_deflate, .lzo, and .bz2).

KeyValueTextInputFormat: This class reads each line and splits the line into a key/value pair on a tab character. The workhorse class is org.apache.hadoop.mapred.
KeyValueLineRecordReader. The separator may be configured by setting the configuration key key.value.separator.in.input.line. The key and value are both Text. If there is no separator found, the value will be an empty string.
NLineInputFormat: This format is ideal for using the input data as control information.
It guarantees that each input split will be N lines long, with one split being the remaining lines. The configuration key mapred.line.input.format.linespermap controls the number of lines of input per map task. The default value is 1. This may be changed using the conf.setInt() method. For example, conf.setInt("mapred.line.
input.format.linespermap", 10) sets the value to 10. Under the covers, this uses org.apache.hadoop.mapred.LineRecordReader to read the input data and produce LongWritable, Text key/value pairs.

TextOutputFormat: This is the standard textual output format. It basically calls the toString method on each key and value, producing a single-line key SEPARATOR value
ASCII newline for each output record. The SEPARATOR is specified by the value of the configuration key apred.textoutputformat.separator, which defaults to TAB. If the value is null, no SEPARATOR and no value will be emitted. If key is null, SEPARATOR value is emitted.
The end-of-record character is hard-coded as an ASCII newline character. Compression is supported if configured.

MultipleTextOutputFormat: This format allows you to write output records to different files based on the key and value. The test case org.apache.hadoop.mapred.
TestMultipleTextOutputFormat provides a sample implementation. The Java source to this class is located in src/test/org/apache/hadoop/mapred/


Comments

Post a Comment