We all know that Hadoop is a framework that enables the distributed processing of large datasets across the clusters of commodity hardware. HDP is HortonWorks Data Platform which is an enterprise grade and hardened hadoop distribution.HDP contains the most useful and stable version of Apache hadoop and its related projects into single tested and certified package.Using HDP, we can install hadoop on windows.

Prerequisites

  • Microsoft Visual C++
  • Microsoft .Net framework4.0
  • java 1.6
  • Python 2.7

Note : After installing java and python , Set environment variables accordingly.

Firewall Configuration

For the communication between Clients and service components, HDP uses multiple ports.So we have to open the ports accordingly.

[code lang="shell"]
netshadvfirewall firewall add rule name=AllowRPCCommunicationdir=in action=allow protocol=TCP localport=$PORT_NUMBER
[/code]

Pre-Installation steps

Create a text file which describes the hadoop components.This text file contains the hostname and other properties of hadoop components.

Note.

  • Ensure that all properties in the text file are separated by the new line character.
  • Ensure that the directory paths do not contain any whitespace character

Example of a text file(here clusterpProperties.txt) file as shown below.

[code lang="shell"]
#Log directory
HDP_LOG_DIR=<Path to Log directory>
#Data directory
HDP_DATA_DIR=<Path to Data directory>
#Hosts
NAMENODE_HOST=<Hostname of namenode machine>
SECONDARY_NAMENODE_HOST=<Hostname of secondary namenode machine>
JOBTRACKER_HOST=<Hostname of Jobtracker machine>
HIVE_SERVER_HOST=<Hostname of Hive server machine>
OOZIE_SERVER_HOST=<Hostname of Oozie server machine>
WEBHCAT_HOST=<Hostname of server machine>
FLUME_HOSTS=<Hostname of server machine>
HBASE_MASTER=<Hostname of Hbase master>
HBASE_REGIONSERVERS=<Hostname of Hbaseregionserver>
ZOOKEEPER_HOSTS=<Hostname of zookeeper machines>
SLAVE_HOSTS=<Hostname of slave machines>
#Database host
DB_HOSTNAME=<Hostname of server machine>
#Hive properties
HIVE_DB_NAME=<Hive Database name>
HIVE_DB_USERNAME=<Database username>
HIVE_DB_PASSWORD=<Database password>
#Oozie properties
OOZIE_DB_NAME=<oozie database name>
OOZIE_DB_USERNAME=<oozie database username&gt>
OOZIE_DB_PASSWORD=<oozie database password&gt>

[/code]

Installation

In command prompt, we have to execute the installation command for HDP

[code lang="shell"]
msiexec/i"<$MSI_PATH>"/lv"<$PATH_to_Installer_Log_File>"
HDP_LAYOUT="<$PATH_to_clusterproperties.txt_File>"
HDP_DIR="<$PATH_to_HDP_Install_Dir>" DESTROY_DATA="<Yes_OR_No>"
[/code]

for example

[code lang="shell"]
msiexec/i"D:\HDPINSTALLATION\hdp-1.3.0.0-GA\hdp-1.3.0.0-GA\hdp-1.3.0.0.winpkg.msi" /lv "D:\HDP\log\installer.log" HDP_LAYOUT="D:\HDPINSTALLATION\hdp-1.3.0.0-GA\hdp-1.3.0.0-GA\clusterproperties.txt" HDP_DIR="D:\HDP\hdp_wd" DESTROY_DATA="no"
[/code]

After the installation, a pop-up window will appear to indicate the installation completion.

Start HDP components

  • cd <Path to HDP directory>
  • start_local_hdp_services

Alternatively we can start these daemons from the services console of windows operating system.

Execute a Mapreduce job

To run a mapreduce job, we have to execute one command ‘Run-SmokeTests.cmd’

  • Run-SmokeTests.cmd