甜橘子，专注成都网站制作网站设计与营销型网站建设与优化
QQ :532337155
电话：028-86922220
这篇文章主要为大家展示了“怎么在Redhat7.4安装CDH5.16.1”，内容简而易懂，条理清晰，希望能够帮助大家解决疑惑，下面让小编带领大家一起研究并学习一下“怎么在Redhat7.4安装CDH5.16.1”这篇文章吧。
集群中各个节点之间能互相通信使用静态IP地址。IP地址和主机名通过/etc/hosts配置，主机名通过/etc/hostname进行配置。
以cm节点（172.31.13.38）为例：
hostname配置
/etc/hostname文件如下：
ip-172-31-13-38.ap-southeast-1.compute.internal
或者你可以通过命令修改立即生效
[root@ip-172-31-13-38 ~]$ hostnamectl set-hostname ip-172-31-13-38.ap-southeast-1.compute.internal
注意：这里修改hostname跟REDHAT6的区别
hosts配置
/etc/hosts文件如下:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 172.31.13.38 ip-172-31-13-38.ap-southeast-1.compute.internal 172.31.11.232 ip-172-31-11-232.ap-southeast-1.compute.internal 172.31.11.9 ip-172-31-11-9.ap-southeast-1.compute.internal 172.31.13.166 ip-172-31-13-166.ap-southeast-1.compute.internal
以上两步操作，在集群中其它节点做相应配置。确认需要安装的4台主机的hosts文件：
在所有节点执行setenforce 0 命令，此处使用批处理shell执行：
[root@ip-172-31-13-38 ~]$ sh ssh_do_all.sh node.list "setenforce 0"
集群所有节点修改/etc/selinux/config文件如下：
SELINUX=disabled SELINUXTYPE=targeted
检查所有机器是否已修改成功：
sh ssh_do_all.sh node.list "cat /etc/selinux/config | grep SELINUX"
集群所有节点执行systemctl stop firewalld命令，此处通过shell批量执行命令如下：
[root@ip-172-31-13-38 ~]$ sh ssh_do_all.sh node.list "systemctl stop firewalld" [root@ip-172-31-13-38 ~]$ sh ssh_do_all.sh node.list "systemctl disable firewalld" [root@ip-172-31-13-38 ~]$ sh ssh_do_all.sh node.list "systemctl status firewalld"
在Redhat7.x的操作系统上，已经默认的安装了chrony，我们这里先卸载chrony，然后安装ntp。使用ntp来配置各台机器的时钟同步，将cm（172.31.13.38）服务作为本地ntp服务器，其它3台服务器与其保持同步。
1.所有机器卸载chrony
[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "yum -y remove chrony"
2.所有机器安装ntp
[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "yum -y install ntp"
3.cm机器配置时钟与自己同步
[root@ip-172-31-16-68 shell]# vim /etc/ntp.conf #server 0.rhel.pool.ntp.org iburst #server 1.rhel.pool.ntp.org iburst #server 2.rhel.pool.ntp.org iburst #server 3.rhel.pool.ntp.org iburst server 127.127.1.0 # local clock fudge 127.127.1.0 stratum 10
4.集群其它节点，配置找cm机器去同步
#server 0.rhel.pool.ntp.org iburst #server 1.rhel.pool.ntp.org iburst #server 2.rhel.pool.ntp.org iburst #server 3.rhel.pool.ntp.org iburst server 172.31.13.38
5.重启所有机器的ntp服务
[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "systemctl restart ntpd" [root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "systemctl status ntpd" [root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "systemctl enable ntpd"
6.验证始终同步，在所有节点执行ntpq -p命令，如下使用脚本批量执行
[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "ntpq -p"
左边出现*号表示同步成功。
所有节点执行
[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "echo vm.swappiness = 10 >> /etc/sysctl.conf" [root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "sysctl vm.swappiness=10"
所有节点执行：
[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/defrag " [root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
设置开机自关闭
将如下脚本添加到/etc/rc.d/rc.local文件中
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi
同步到所有节点
Fayson用的是AWS的环境，这步是可以省略的，放在这里供物理机部署的兄弟们参考。
挂载操作系统iso文件
[root@ip-172-31-2-159 ~]$ sudo mkdir /media/DVD1 [root@ip-172-31-2-159 ~]$ sudo mount -o loop CentOS-7-x86_64-DVD-1611.iso /media/DVD1/
配置操作系统repo
[root@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/local_os.repo [local_iso] name=CentOS-$releasever - Media baseurl=file:///media/DVD1 gpgcheck=0 enabled=1 [root@ip-172-31-2-159 ~]$ sudo yum repolist
安装httpd服务
[root@ip-172-31-2-159 ~]$ yum -y install httpd
启动或停止httpd服务
[root@ip-172-31-2-159 ~]$ systemctl start httpd [root@ip-172-31-2-159 ~]$ systemctl stop httpd
安装完httpd后，重新制作操作系统repo，换成http的方式方便其它服务器也可以访问
[root@ip-172-31-2-159 ~]$ sudo mkdir /var/www/html/iso [root@ip-172-31-2-159 ~]$ sudo scp -r /media/DVD1/* /var/www/html/iso/ [root@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/os.repo [osrepo] name=os_repo baseurl=http://172.31.2.159/iso/ enabled=true gpgcheck=false [root@ip-172-31-2-159 ~]$ sudo yum repolist
1.安装MariaDB
[root@ip-172-31-13-38 ~]# yum -y install mariadb [root@ip-172-31-13-38 ~]# yum -y install mariadb-server
2.启动并配置MariaDB
[root@ip-172-31-13-38 ~]# systemctl start mariadb [root@ip-172-31-13-38 ~]# systemctl enable mariadb [root@ip-172-31-13-38 ~]# /usr/bin/mysql_secure_installation NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! In order to log into MariaDB to secure it, we'll need the current password for the root user. If you've just installed MariaDB, and you haven't set the root password yet, the password will be blank, so you should just press enter here. Enter current password for root (enter for none): OK, successfully used password, moving on... Setting the root password ensures that nobody can log into the MariaDB root user without the proper authorisation. Set root password? [Y/n] Y New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! By default, a MariaDB installation has an anonymous user, allowing anyone to log into MariaDB without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? [Y/n] Y ... Success! Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] n ... skipping. By default, MariaDB comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] Y - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] Y ... Success! Cleaning up... All done! If you've completed all of the above steps, your MariaDB installation should now be secure. Thanks for using MariaDB!
3.建立CM，Hive等需要的表
[root@ip-172-31-13-38 ~]# mysql -u root -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 9 Server version: 5.5.56-MariaDB MariaDB Server Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> create database metastore default character set utf8; CREATE USER 'hive'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%'; FLUSH PRIVILEGES; create database cm default character set utf8; CREATE USER 'cm'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%'; FLUSH PRIVILEGES; create database am default character set utf8; CREATE USER 'am'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON am. * TO 'am'@'%'; FLUSH PRIVILEGES; create database rm default character set utf8; CREATE USER 'rm'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%'; FLUSH PRIVILEGES; create database hue default character set utf8; CREATE USER 'hue'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%'; FLUSH PRIVILEGES; create database oozie default character set utf8; CREATE USER 'oozie'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%'; FLUSH PRIVILEGES; create database sentry default character set utf8; CREATE USER 'sentry'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON sentry. * TO 'sentry'@'%'; FLUSH PRIVILEGES; create database nav_ms default character set utf8; CREATE USER 'nav_ms'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON nav_ms. * TO 'nav_ms'@'%'; FLUSH PRIVILEGES; create database nav_as default character set utf8; CREATE USER 'nav_as'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON nav_as. * TO 'nav_as'@'%'; FLUSH PRIVILEGES;
安装jdbc驱动
[root@ip-172-31-13-38 ~]# mkdir -p /usr/share/java/ [root@ip-172-31-13-38 ~]# mv mysql-connector-java-5.1.34.jar /usr/share/java/ [root@ip-172-31-13-38 ~]# cd /usr/share/java [root@ip-172-31-13-38 java]# ln -s mysql-connector-java-5.1.34.jar mysql-connector-java.jar [root@ip-172-31-13-38 java]# ll total 940 -rwxrwxr-x. 1 root root 960372 Feb 1 08:31 mysql-connector-java-5.1.34.jar lrwxrwxrwx 1 root root 31 Feb 2 00:52 mysql-connector-java.jar -> mysql-connector-java-5.1.34.jar
1.下载CM5.16.1的安装包，地址为：
http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/enterprise-debuginfo-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/jdk-6u31-linux-amd64.rpm http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
2.下载CDH5.16.1的安装包，地址为：
http://archive.cloudera.com/cdh6/parcels/5.16.1/CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel http://archive.cloudera.com/cdh6/parcels/5.16.1/CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha1 http://archive.cloudera.com/cdh6/parcels/5.16.1/manifest.json
3.将Cloudera Manager安装需要的7个rpm包下载到本地，放在同一目录，执行createrepo命令生成rpm元数据。
[root@ip-172-31-13-38 cm5.16.1]# ll total 1019160 -rw-r--r-- 1 root root 9864584 Nov 27 14:40 cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm -rw-r--r-- 1 root root 789872988 Nov 27 14:40 cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm -rw-r--r-- 1 root root 8704 Nov 27 14:40 cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm -rw-r--r-- 1 root root 10612 Nov 27 14:40 cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm -rw-r--r-- 1 root root 30604172 Nov 27 14:40 enterprise-debuginfo-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm -rw-r--r-- 1 root root 71204325 Nov 27 14:40 jdk-6u31-linux-amd64.rpm -rw-r--r-- 1 root root 142039186 Nov 27 14:40 oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm [root@ip-172-31-13-38 cm5.16.1]# createrepo . Spawning worker 0 with 2 pkgs Spawning worker 1 with 2 pkgs Spawning worker 2 with 2 pkgs Spawning worker 3 with 1 pkgs Workers Finished Saving Primary metadata Saving file lists metadata Saving other metadata Generating sqlite DBs Sqlite DBs complete
4.配置Web服务器
将上述cdh6.16.1/cm5.16.1目录移动到/var/www/html目录下, 使得用户可以通过HTTP访问这些rpm包。
[root@ip-172-31-13-38 ~]# mv cm5.16.1/ cdh6.16.1/ /var/www/html/
验证浏览器能否正常访问
5.制作Cloudera Manager的repo源
[root@ip-172-31-13-38 ~]# vim /etc/yum.repos.d/cm.repo [cmrepo] name = cm_repo baseurl = http://172.31.13.38/cm5.16.1 enable = true gpgcheck = false [root@ip-172-31-13-38 yum.repos.d]# yum repolist Loaded plugins: amazon-id, rhui-lb, search-disabled-repos repo id repo name status cmrepo cm_repo 7 rhui-REGION-client-config-server-7/x86_64 Red Hat Update Infrastructure 2.0 Client C 1 rhui-REGION-rhel-server-releases/7Server/x86_64 Red Hat Enterprise Linux Server 7 (RPMs) 20,668 rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Commo 233 repolist: 20,909
6.验证安装JDK
yum -y install oracle-j2sdk1.7-1.7.0+update67-1
1.通过yum安装Cloudera Manager Server
yum -y install cloudera-manager-server
2.初始化数据库
[root@ip-172-31-13-38 yum.repos.d]# /usr/share/cmf/schema/scm_prepare_database.sh mysql cm cm password JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.7.0_67-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. [ main] DbCommandExecutor INFO Successfully connected to database. All done, your SCM database is configured correctly!
3.启动Cloudera Manager Server
[root@ip-172-31-13-38 ~]# systemctl start cloudera-scm-server
4.检查端口是否监听
[root@ip-172-31-13-38 ~]# netstat -lnpt | grep 7180 tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 3331/java
5.通过http://13.250.14.115:7180/cmf/login访问CM
1.admin/admin登录到CM
2.同意license协议，点击继续
3.选择60试用，点击继续
4.点击“继续”
5.输入主机ip或者名称，点击搜索找到主机后点击继续
6.点击“继续”
注意这里列出来的CDH版本都是系统最开始默认的，来自于Cloudera公网的下载源仓库，这里我们需要先将CDH的安装源修改一下。
7.使用parcel选择，点击“更多选项”,点击“-”删除其它所有地址，输入
http://172.31.13.38/cdh6.16.1，点击“保存更改”
保存更改后，这时回到上个页面会看到我们之前准备好的http的CDH5.16.1的源，如果显示不出来，可能http源配置有问题，请参考前面步骤仔细进行检查。
8.选择自定义存储库，输入cm的http地址
9.点击“继续”，进入下一步安装jdk
10.点击“继续”，进入下一步，默认多用户模式，不需要进行任何勾选
11.点击“继续”，进入下一步配置ssh账号密码
12.点击“继续”，进入下一步，安装Cloudera Manager相关到各个节点
13.点击“继续”，进入下一步安装cdh到各个节点
14.点击“继续”，进入下一步主机检查，确保所有检查项均通过
点击完成进入服务安装向导。
1.选择需要安装的服务
2.点击“继续”，进入集群角色分配
3.点击“继续”，进入下一步，测试数据库连接
4.测试成功，点击“继续”，进入目录设置，此处使用默认默认目录，根据实际情况进行目录修改
5.点击“继续”，进入各个服务启动
6.安装成功
7.安装成功后进入CM管理界面主页
CDH从5.13开始，直接打包Kudu，不再需要下载额外的Parcel和csd文件。CDH5.13集成的是Kudu1.5，CDH5.16.1集成的是Kudu1.7。
1.通过CM安装Kudu1.7
添加Kudu服务
2.选择Master和Tablet Server
3.配置相应的目录，注：无论是Master还是Tablet根据实际情况数据目录(fs_data_dir)应该都可能有多个，以提高并发读写，从而提高Kudu性能。
4.启动Kudu服务
5.安装完毕
6.配置Impala
回到主页重启Impala和Hue
至此，CDH5.16.1+Kudu1.7安装完毕。
以上是“怎么在Redhat7.4安装CDH5.16.1”这篇文章的所有内容，感谢各位的阅读！相信大家都有了一定的了解，希望分享的内容对大家有所帮助，如果还想学习更多知识，欢迎关注创新互联行业资讯频道！
另外有需要云服务器可以了解下创新互联scvps.cn，海内外云服务器15元起步，三天无理由+7*72小时售后在线，公司持有idc许可证，提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案，具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势，专为企业上云打造定制，能够满足用户丰富、多元化的应用场景需求。