IP address	hostname (of a networked computer)	instructions
192.168.75.41	cdh01	Master node, slave node
192.168.75.42	cdh02	from a node
192.168.75.43	cdh03	from a node

II. commencement erect formerly configure and pre-installed software

1、 server configure

Modify the per-node server about the configure， packet includeIP、hostname、selinux， firewall。

If you are in erectCentOS7 The system has been specified at the time of hostname (of a networked computer) harmony IP address， or so IP Modification harmony hostname change Then you can omit。 But the point to emphasize here is， If you want to use theyum erect， or soDNS must configure。

1． IP Modification

Here to use the fixedIP， but (not)centOS 7 Not anymore.setup command， So the GUI settings are no longer available， Here you can only modify configure documents， moreover configure The name of the document is not yet fixed， Everyone's can be different.， Here I'll say the location，IP configure Location of the document：/etc/sysconfig/network-scripts/， as shown below：

Here's mine. configure The name of the file is calledifcfg-ens33。 but (not)CentOS 7 It's amazing that the system evenvim And no more.， onlyvi， So it's the following：

as shown above：BOOTPROTO、ONBOOT It needs to be modified.， The remaining four items are to be added manually。 Because later on you have to useyum erect， So here it must be configureDNS，test (machinery etc) The environment can be madeDNS configure into the address of the gateway。

Restart the network service as shown above, this command was not changed, then check the ip.

Each unit performs such configure。

2． hostname change

Make changes to all three separately, and note the name and ip of each one, matching each one.

[root@hodoop1~] vi /etc/sysconfig/network
# configure elements
NETWORKING=yes
HOSTNAME=cdh01

This change will require a reboot to take effect, or it can take effect immediately by using the following.

[root@hodoop1~] hostname cdh01
[root@hodoop1~] hostname
cdh01

3． Turn off the firewall

CentOS 7.0 uses firewall by default, and the related operation has changed a lot, and is not at all the same as in 6.5, as follows.

 [root@hadoop1~]firewall-cmd --state #View firewall status
running
 [root@hadoop1~]systemctl stop firewalld # Stop the firewall's services
 [root@hadoop1~]systemctl disable firewalld # Disable bootup
rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'
rm '/etc/systemd/system/basic.target.wants/firewalld.service'

4． selinux shutdown

All nodes need to be closed, as required by the official documentation.

[root@hadoop1~]vi /etc/sysconfig/selinux
SELINUX=disabled

Reboot to take effect, check after reboot.

[root@cdh01~]sestatus -v
 SELinux status: disabled # indicates that it has been turned off.

5． No Secret Login configure

erect in the course ofmaster The need for individual nodes of theroot No login password， first inmaster Generate public key on：

ssh-keygen
ssh-copy-id root@192.168.75.41
ssh-copy-id root@192.168.75.42
ssh-copy-id root@192.168.75.43

Do the above for each node separately and verify that you can log in to the node server directly without entering a password by using the ssh ip when you are done.

6．ip harmony hostname (of a networked computer) mapping relationship

The operation is as follows.

[root@hadoop1~]vi /etc/hosts
# It reads as follows
127.0.0.1 localhost
::1 localhost
192.168.75.41 cdh01
192.168.75.42 cdh02
192.168.75.43 cdh03

Content as above， Save to exit， Then you can use thescp The command will take this configure Copy to other hosts。

scp /etc/hosts root@192.168.75.42:/etc/hosts

7． NTP server settings

Many blogs online mention this configure， It says to make the nodes uniform in time， But I didn't. configure， I'm also putting this configure Put it here.：

 [root@cdh01~]yum -y install ntp #Change the master's node
[root@cdh01~]vi /etc/ntp.conf

Comment out all server *.*.* pointers and add a new connectable ntp server (Baidu for ntp server)

server ntp.sjtu.edu.cn iburst

Just point ntp to the master server address on the other nodes (under /etc/ntp.conf)

server 192.168.160.130 iburst All nodes.

2、 Third-party reliance packet

1． Other dependencies

For this step， You can read many blogs online that put this step intoMySQL erect behind the， But after my owntest (machinery etc)， It's better to put this step in front， Because behind erect Software， But where there is dependence， You've done it here， So operate here erect depend packet， And it's all nodes erect。

yum install chkconfig python bind-utils psmisc libxslt zlib sqlite fuse fuse-libs redhat-lsb cyrus-sasl-plain cyrus-sasl-gssapi

Note that this place dependency package must be installed completely, will go through three y/N process, all choose Y, this installation process depends entirely on your Internet speed, basically after knocking this command you can rest for a while.

2．MySQL jar packet

This session only needs to be done on the master node.

atcdh01 on preparationmysql ofjar packet：

mkdir -p /usr/share/java

revisejar packet name， and copy to/usr/share/java/ directory：

cp mysql-connector-java-5.1.38-bin.jar /usr/share/java/mysql-connector-java.jar

Note the above， It's modifiedjar packet name， Because there's an operation behind it that looks for thisjar packet this fixed name， So here's a change of name。

3、 erectjdk

This is the foundation of all services， Each node needs erect。

1． uninstallationjdk

erect Check it first before， Does your server default to erect finishOpenJDK， If the system comes with， would require uninstallation， The steps are as follows：

# Check if erect
[root@cdh01~]java -version
java version "1.7.0_75"
OpenJDK Runtime Environment (rhel-2.5.4.2.el7_0-x86_64 u75-b13)
OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)
# look over need uninstallation of packet
[root@cdh01~]rpm -qa | grep jdk
java-1.7.0-openjdk-1.7.0.75-2.5.4.2.el7_0.x86_64
java-1.7.0-openjdk-headless-1.7.0.75-2.5.4.2.el7_0.x86_64
# uninstallation
[root@cdh01~]yum -y remove java-1.7.0-openjdk-1.7.0.75-2.5.4.2.el7_0.x86_64
[root@cdh01~]yum -y remove java-1.7.0-openjdk-headless-1.7.0.75-2.5.4.2.el7_0.x86_64
# Double check.
[root@cdh01~]java -version
bash: /usr/bin/java: No such file or directory

If not erect， You can skip this step， My system doesn't come with itjdk。

2． erect

Choose either of the two methods below, don't do both.

1> rpm or yum installation

Use commands directly erect Can， There's not much to explain here。

2> Use compression packet erect

Upload the prepared installation package, the directory I'm used to is: /home/software, finish uploading and unzip:.

tar -zxvf jdk-8u131-linux-x64.tar.gz
mv jdk-1.8.0 jdk1.8

If you want to erectspark， You can combine the environment variables withScala together configure。

configure environment variable：

vi /etc/profile

Add the following to the document.

#java
export JAVA_HOME=/home/software/jdk1.8
export PATH=$PATH:$JAVA_HOME/bin

After saving the exit, a soft connection needs to be set up here as well:.

mkdir /usr/java
ln -s /home/software/jdk1.8 /usr/java/default

If you are usingyum orrpm erect ofjdk These steps can then be omitted。 BecauseCDH platform erect The default look for when thejdk The path is/usr/java。

4、 erectScala

If you want to erectSpark Must not be omittedScala of erect， if not， You're in.web end-to-end operationCDH when， will give you a mistake， The solution on the web， It's basically all saidjdk No one said an ever-say-wellScala。

This operation is likewise on all nodes.

The directory is still there/home/software。 The operation is as follows.

tar -zxvf scala-2.11.0.tgz
mv scala-2.11.0 scala2.11

And then configure environment variable：

vi /etc/profile

Also add the following at the end.

export SCALA_HOME=/home/software/scala2.11
export PATH=$PATH:$SCALA_HOME/bin

5、 erectmysql

MySQL installation only needs to be done on the master node.

1． uninstallation

erectMySQL It also needs to be checked， Whether the system comes with it or not，centos7 Comes with itmariadb， Check first， The command is as follows：

[root@cdh01]rpm -qa | grep mariadb
mariadb-libs-5.5.41-2.el7_0.x86_64

A situation such as the above exists， If present use the following command for uninstallation：

[root@cdh01]rpm -e --nodeps mariadb-libs-5.5.41-2.el7_0.x86_64

2． erect

Place the downloadedMySQLrpm packet Copy to the server， Then unzip it.、 erect。

1> Creating Users and User Groups

 # Adding the user group mysql.
groupadd mysql
 #Add user mysql, add mysql user group.
useradd -r -g mysql mysql

2> erectMySQL

# erectserver：
rpm -ivh MySQL-server-5.6.29-1.linux_glibc2.5.x86_64.rpm
# erectclient：
rpm -ivh MySQL-client-5.6.29-1.linux_glibc2.5.x86_64.rpm

3> Add random start

Add mysqld to the system service and start it randomly with the following command.

cp /usr/share/mysql/mysql.server /etc/init.d/mysqld

4> Starting MySQL

The command to start mysqld is as follows.

service mysqld start

5> Change Password

First get the random password for the root user on the mysql installation:.

vim /root/.mysql_secret

It can also be viewed using the cat command at

cat /root/.mysql_secret

This password can only be used to change the password.

You must change the password of the root user to use mysql, otherwise you can only connect but not operate

mysqladmin -u root -p password root

6>test (machinery etc)

 #connect into mysql with the following command.
mysql -u root -p
root
# look overmysql of erect Run path， The command is as follows：
ps -ef|grep mysql

7> MySQL Related Questions

If there is a problem with no permissions, authorize in mysql authorization (executed on the machine where mysql is installed) by executing the following statement.

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'root' WITH GRANT OPTION;
FLUSH PRIVILEGES;

*.*:All tables under all libraries.

%： any IP address or the host can connect。 If% configure Does not take effect， only configure concrete hostname (of a networked computer) call。

3． Create database

The following command is to create the database needed to deploy each service, I personally prefer not to create a good first, when you can directly deploy the service, do not have to come back to the database to create.

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

III. erectCloudera-Manager

willCloudera Related to the four packet， as follows：

cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz

CDH-5.7.2-1.cdh5.7.2.p0.11-el7.parcel

CDH-5.7.2-1.cdh5.7.2.p0.11-el7.parcel.sha1

manifest.json

Upload to all servers and perform the following actions on all servers.

1、 erectCM

unpackcm tar packet to the specified directory， first establish directory， command The operation is as follows.

[root@cdh01 ~]mkdir /opt/cloudera-manager
[root@cdh01 ~]tar -zxvf cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz -C /opt/cloudera-manager

Create cloudera-scm user

# establish
[root@cdh01 ~]useradd -r -d /opt/cloudera-manager/cm-5.7.2/run/cloudera-scm-server -M -c "Cloudera SCM User" cloudera-scm
# look over
[root@cdh01 ~]id cloudera-scm

2、 configure

1． configure from a node leader of a group

configure from a nodecloudera-manger-agent Pointing to the master node server， My current cluster planning， is required on each server as follows configure：

vi /opt/cloudera-manager/cm-5.7.2/etc/cloudera-scm-agent/config.ini

willserver_host change intoCMS whereabouts hostname (of a networked computer) namelycdh01

server_host=cdh01

2． configure Warehouse catalog

This operation can be performed only on the master node, where the parcel-repo repository directory is created with the following command.

[root@cdh01 ~]mkdir -p /opt/cloudera/parcel-repo
[root@cdh01 ~]chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
[root@cdh01 ~]cp CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha1 manifest.json /opt/cloudera/parcel-repo

Note: where CDH-5.7.2-1.cdh5.7.2.p0.18-el5.parcel.sha1 suffix is to be removed from 1, the operation is as follows.

mv CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha1 CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha

3． configureCDH from a node directory

Create the parcels directory on all nodes by doing the following.

mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

Explanation: Clouder-Manager extracts CDH from the /opt/cloudera/parcel-repo directory of the master node and distributes the decompression activation to the /opt/cloudera/parcels directory of each node.

4． Initializing the database

This operation is performed on the master node with the initial script configure database scm_prepare_database.sh with the following action command.

[root@cdh01 ~]/opt/cloudera-manager/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql -hcdh01 -uroot -proot --scm-host cdh01 scmdbn scmdbu scmdbp

Description: This script is the one used to create and configure the database needed by the CMS. The parameters are.

mysql： The database is usedmysql， If erect Used in the processoracle， Then the argument should be changed tooracle。

-hcdh01: The database is built on top of the cdh01 host. That is, above the master node.

-uroot: run mysql as root. -proot: the root password for mysql is root.

--scm-host cdh01：CMS Host， Generally it's a summysql erect The host of is on the same host。

The last three parameters are: database name, database username, and database password.

Execution of the completion order is normal as follows.

This is where it explains why the jar name was changed above.

3、 Start Service

1． Start the server

This command can be executed only on the master node.

Go to the /opt/cloudera-manager/cm-5.7.2/etc/init.d/ directory and execute the following command.

./cloudera-scm-server start
 # Or simply execute the following command.
/opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-server start

2． Start agent

Start cloudera-scm-agent, which needs to be started on all slave nodes.

Go to the /opt/cloudera-manager/cm-5.7.2/etc/init.d/ directory and execute the following command.

./cloudera-scm-agent start
 # Or simply execute the following command.
/opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-agent start

Activation is shown in the following figure.

Just to clarify here, basically both of the above starts will be configured to randomly start, but I experimented with many ways to randomly start and none of them worked, so I'll leave it here for now and add it later when I find a way to randomly start.

Note: The following steps will not cause any problems as long as the above links are done without any problems.

IV. serve erect

1、 web login

Type 192.168.75.41:7180 into your browser.

The following login screen appears with the default username and password: admin

This interface appears instructionsCM already erect It worked， Here's itweb Various services are deployed in the interface。

2、 web boot installation

No screens are mentioned in this process, and continue is selected by default.

1． Select the express version first

When logged in, you will be taken to the screen to select the express version, just select free on this screen and continue.

2． configure host

Each of the servers from a node already erect and startedagent， Also on each node configure finishserver point to， So each node'sagent will giveserver Send a message to report， It is possible to find it here in“ Currently managed hosts” Three hosts are seen in， Check all and continue。

Note that ifcloudera-scm-agent Not set to boot up， If the above has restarted here it may not detect other servers。 before configure when， I didn't configure Random start。

3． Select CDH version

Just choose the version you need here. Here I have chosen two items, one of the selection methods uses Parcel; the second CDH version selected is CDH-5.7.2-1 cdh5.7.2p0 18; all others are selected as none.

4． erectParcel

This step is to distribute parcels to each node, the previous step continues, this step is automatic no action is needed, the only thing needed is to wait. When all are done click Continue.

5． configure verify

This step is for the host configure the correctness of the test。

This is a place to note that two items in this area didn't pass the check, but I didn't do anything about it, I found a method online, but I don't know if it works exactly, in the following way.

The following commands can be used in a cluster

echo 0 > /proc/sys/vm/swappiness
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Then clicking re-run above will show that all checks passed this time.

I didn't do the above, but the back install didn't affect it either.

6． Select Service

The play is coming， Here's what you want to choose erect The service used，CDH5 By default, six combinations are available as shown， Of course, that can also be customized， Here I choose customization。

7． Role assignment

Here's the service Role assignment To that node， It can be defaulted to， Or you can make your own choices。

8． Database settings selection

This is where the individual databases created earlier are used, and depending on the service you choose, this will ask you to fill in the database used for each service, as well as the username and password.

9． Cluster Audit

Just leave it all as default here, unless you have a plan and can make changes here.

10． commencement erect

If web boot installation Something could go wrong.， Then the problem is all in this step， Questions that will arise here， The problem I'm having， It's basically all described earlier， It is erectjdk harmonyScala when。 Basically nothing else will go wrong。

11． erect finish

At this point the installation is complete, click Finish and you will be taken to the following screen.

Checking the cluster situation, I have quite a lot of alarms here, probably looking at them is basically alarms for memory or storage usage thresholds, since I'm a local VM, these conditions are limited, so I won't deal with them here for now.

That's all erect the whole process。

Recommended>>
1、Learning python from scratch section 102 HTTP request data
2、From Bitcoin to blockchain why blockchain can only be a sidechain to cryptocurrency chains
3、The first artificial intelligence mobile phone today down a thousand dollars good value configuration general will usher in a new system
4、Hundreds of thousands of desktop wallpapers to choose from Its completely free Play whatever you want
5、VIIIInfosphereDataReplicationCDCUserExit Example