NT1330 Unit 1 Assignment Analysis
Assignment No: 3
Title : Programming/Configuration of project functions as per UML diagram mention in earlier submission.
Objectives :
Now third assignment is all about programming/configuration of project functions as per the UML diagram mention in earlier submission. softwares required:
1) jdk 1.8(32 bit)
2) eclipse-luna
3) Apache Tomcat
HADOOP MULTI NODE INSTALLATION
HADOOP INSTALLATION
Hadoop requires a working Java 1.5+ (aka Java 5) installation.
Update the source list user@ubuntu:~$ sudo apt-get update or
Install Sun Java 6 JDK Note:
If you already have Java JDK installed on your system, then you need not run the above command.
To install it user@ubuntu:~$ sudo apt-get install sun-java6-jdk
The full JDK which will be placed in …show more content…
Master as well as slave)
fs.default.name hdfs://master:54310 The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.
conf/mapred-site.xml
Open this file in the conf directory hduser@master:~$ /usr/local/hadoop/conf hduser@master:~$ sudo gedit mapred-site.xml
Change the mapred.job.tracker parameter (in conf/mapred-site.xml), which specifies the JobTracker (MapReduce master) host and port. conf/mapred-site.xml (ALL machines)
mapred.job.tracker master:54311 The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task.
conf/hdfs-site.xml
Open this file in the conf directory hduser@master:~$ /usr/local/hadoop/conf hduser@master:~$ sudo gedit hdfs-site.xml
Change the dfs.replication parameter (in conf/hdfs-site.xml) which specifies the default block replication. We have two nodes available, so we set dfs.replication to …show more content…
The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.
Formatting the HDFS filesystem via the NameNode
Format the cluster’s HDFS file system hduser@master:~/usr/local/hadoop$ bin/hadoop namenode -format Starting the multi-node cluster
Starting the cluster is performed in two steps.
1. We begin with starting the HDFS daemons: the NameNode daemon is started on master, and DataNode daemons are started on all slaves (here: master and slave).
2. Then we start the MapReduce daemons: the JobTracker is started on master, and TaskTracker daemons are started on all slaves (here: master and slave).
Cluster is started by running the commnd on master hduser@master:~$ /usr/local/hadoop
hduser@master:~$
Title : Programming/Configuration of project functions as per UML diagram mention in earlier submission.
Objectives :
Now third assignment is all about programming/configuration of project functions as per the UML diagram mention in earlier submission. softwares required:
1) jdk 1.8(32 bit)
2) eclipse-luna
3) Apache Tomcat
HADOOP MULTI NODE INSTALLATION
HADOOP INSTALLATION
Hadoop requires a working Java 1.5+ (aka Java 5) installation.
Update the source list user@ubuntu:~$ sudo apt-get update or
Install Sun Java 6 JDK Note:
If you already have Java JDK installed on your system, then you need not run the above command.
To install it user@ubuntu:~$ sudo apt-get install sun-java6-jdk
The full JDK which will be placed in …show more content…
Master as well as slave)
fs.default.name hdfs://master:54310 The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.
conf/mapred-site.xml
Open this file in the conf directory hduser@master:~$ /usr/local/hadoop/conf hduser@master:~$ sudo gedit mapred-site.xml
Change the mapred.job.tracker parameter (in conf/mapred-site.xml), which specifies the JobTracker (MapReduce master) host and port. conf/mapred-site.xml (ALL machines)
mapred.job.tracker master:54311 The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task.
conf/hdfs-site.xml
Open this file in the conf directory hduser@master:~$ /usr/local/hadoop/conf hduser@master:~$ sudo gedit hdfs-site.xml
Change the dfs.replication parameter (in conf/hdfs-site.xml) which specifies the default block replication. We have two nodes available, so we set dfs.replication to …show more content…
The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.
Formatting the HDFS filesystem via the NameNode
Format the cluster’s HDFS file system hduser@master:~/usr/local/hadoop$ bin/hadoop namenode -format Starting the multi-node cluster
Starting the cluster is performed in two steps.
1. We begin with starting the HDFS daemons: the NameNode daemon is started on master, and DataNode daemons are started on all slaves (here: master and slave).
2. Then we start the MapReduce daemons: the JobTracker is started on master, and TaskTracker daemons are started on all slaves (here: master and slave).
Cluster is started by running the commnd on master hduser@master:~$ /usr/local/hadoop
hduser@master:~$