1. 实验环境
节点 1:
节点 2:
v1.shenmin.com
v2.shenmin.com
192.168.137.11
192.168.137.12
iscsi 存储/配置服务器:
rhel6.shenmin.com
192.168.137.6
1.1. Hosts 文件或 DNS 设置
修改/etc/hosts 文件或者搭建 DNS 服务器,三台 server 的 hosts 文件如下:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.137.11 v1.shenmin.com v1
192.168.137.12 v2.shenmin.com v2
192.168.137.6 rhel6.shenmin.com rhel6
1.2. 配置使用 DVD 做 yum 源
1. 新建 dvd 挂载目录
[root@rhel6 ~]# mkdir /media/iso
2. 挂载 dvd 到指定目录
[root@rhel6 ~]# mount /dev/cdrom /media/iso/
3. 进入 yum.repos.d 文件夹,
[root@rhel6 ~]# cd /etc/yum.repos.d/
4. 删除原有的 repo 文件
[root@rhel6 yum.repos.d]# rm -rf *
5. 新建使用 dvd 为源的 repo 文件
[root@rhel6 yum.repos.d]# vim iso.repo
[Server]
name=Server
baseurl=file:///media/iso/Server
enabled=1
gpgcheck=0
[HighAvailability]
name=HighAvailability
baseurl=file:///media/iso/HighAvailability
enabled=1
gpgcheck=0
[LoadBalancer]
name=LoadBalancer
baseurl=file:///media/iso/LoadBalancer
enabled=1
gpgcheck=0
[LoadBalancer]
name=LoadBalancer
baseurl=file:///media/iso/LoadBalancer
enabled=1
gpgcheck=0
6. 清空原有 yum 信息
[root@rhel6 yum.repos.d]# yum clean all
7. 更新 yum
[root@rhel6 yum.repos.d]# yum update
2. 配置集群之前
2.1. 存储配置
2.1.1. ISCSI 服务器
1.
rhel6.shenmin.com 上 sda1 和 sda2 为/分区和 swap 分区,尚余一部分磁盘空间未分配,
用 fdisk 分出 sda3 一个分区(10G)准备用作 iscsi 共享存储。
[root@rhel6 ~]# fdisk -l
Disk /dev/sda: 320.1 GB, 320071851520 bytes
64 heads, 32 sectors/track, 305244 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x49e2fd2f
Device Boot Start End Blocks Id System
/dev/sda1 * 2 102401 104857600 83 Linux
/dev/sda2 102402 106497 4194304 82 Linux swap / Solaris
/dev/sda3 106498 116738 10486784 83 Linux
2. 安装 scsi-target-utils 软件提供 iscsi 服务
[root@rhel6 ~]# yum install scsi-target-utils
3. 修改/etc/tgt/targets.conf,添加:
backing-store /dev/sda3
这只是最简单的配置,其他设定请参考 target.cong 中的注释说明
4. 重启 tgtd 服务,并将其设为开机启动:
[root@rhel6 ~]# /etc/init.d/tgtd restart
停止 SCSI target daemon:not running [失败]
正在启动 SCSI target daemon: [确定]
[root@rhel6 ~]# chkconfig tgtd on
[root@rhel6 ~]# chkconfig tgtd --list
tgtd 0:关闭 1:关闭 2:启用 3:启用 4:启用 5:启用 6:关闭
5. 查看服务状态,可以看到已经提供了一个 10G 的 ISCSI LUN 1 卷
[root@rhel6 ~]# tgt-admin --show
TTarget 1: iqn.2011-06.com.domain:server.target1
System information:
Driver: iscsi
State: ready
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 10738 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sda3
Backing store flags:
Account information:
ACL information:
ALL
6. 打开防火墙端口 3260 或者关闭防火墙,具体设置自行解决。
2.1.2. ISCSI 客户机
1. 安装 iscsi 客户端软件 iscsi-initiator-utils
[root@v1 ~]# yum install iscsi-initiator-utils
2. 发现服务器 rhel6.shenmin.com 的 iscsi 共享卷
[root@v1 ~]# iscsiadm -m discovery -t sendtargets -p rhel6
192.168.137.6:3260,1 iqn.2011-06.com.domain:server.target1
[root@v2 ~]# iscsiadm -m discovery -t sendtargets -p 192.168.137.6
192.168.137.6:3260,1 iqn.2011-06.com.domain:server.target1
Rhel6 为主机名,使用 ip 地址也可
3. 登录 ISCSI 存储:
[root@v1 ~]# iscsiadm -m node -T iqn.2011-06.com.domain:server.target1 -p
192.168.137.6:3260 -l
Logging in to [iface: default, target: iqn.2011-06.com.domain:server.target1,
portal: 192.168.137.6,3260]
Login to [iface: default, target: iqn.2011-06.com.domain:server.target1, portal:
192.168.137.6,3260] successful.
使用 iscsiadm -m node -T target 名称 -p targetIP 地址:端口号 –login 命令
4. 查看本地存储,可以发现/dev/sda
[root@v1 ~]# fdisk -l
Disk /dev/vda: 8589 MB, 8589934592 bytes
16 heads, 63 sectors/track, 16644 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00089256
Device Boot Start End Blocks Id System
/dev/vda1 * 3 1018 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/vda2 1018 16645 7875584 8e Linux LVM
Partition 2 does not end on cylinder boundary.
Disk /dev/sda: 10.7 GB, 10738466816 bytes
64 heads, 32 sectors/track, 10241 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sda doesn't contain a valid partition table
5. 卸载 ISCSI 存储
[root@v1 ~]# iscsiadm -m node -T iqn.2011-06.com.domain:server.target1 -p
192.168.137.6:3260 -u
Logging out of session [sid: 1, target: iqn.2011-06.com.domain:server.target1,
portal: 192.168.137.6,3260]
Logout of [sid: 1, target: iqn.2011-06.com.domain:server.target1, portal:
192.168.137.6,3260] successful.
6. 再次查看本地存储,/dev/sda 已经卸载
[root@v1 ~]# fdisk -l
Disk /dev/vda: 8589 MB, 8589934592 bytes
16 heads, 63 sectors/track, 16644 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00089256
Device Boot Start End Blocks Id System
/dev/vda1 * 3 1018 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/vda2 1018 16645 7875584 8e Linux LVM
Partition 2 does not end on cylinder boundary.
7. 在其他的节点上挂载 ISCSI,可以看到:
[root@v2 ~]# fdisk -l
Disk /dev/vda: 8589 MB, 8589934592 bytes
16 heads, 63 sectors/track, 16644 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00089256
Device Boot Start End Blocks Id System
/dev/vda1 * 3 1018 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/vda2 1018 16645 7875584 8e Linux LVM
Partition 2 does not end on cylinder boundary.
Disk /dev/sda: 10.7 GB, 10738466816 bytes
64 heads, 32 sectors/track, 10241 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sda doesn't contain a valid partition table
2.2. 集群软件包安装
分为集群节点和配置管理器(Luci,RHCS 的 web 管理工具,仅仅只是配置工具,集群
可脱离它正常运行)
2.2.1. 集群节点
1. 配置好 yum 源后,安装集群软件包
[root@v1 ~]# yum install ricci openais cman rgmanager lvm2-cluster gfs2-utils
2. 打开防火墙相应端口
端口
5404,5405
11111
21064
16851
协议
UDP
TCP
TCP
TCP
程序
corosync/cman
ricci
dlm
modclusterd
3. 或者直接禁用防火墙
[root@v1 ~]# /etc/init.d/iptables stop
iptables:清除防火墙规则: [确定]
iptables:将链设置为政策 ACCEPT:filter [确定]
iptables:正在卸载模块: [确定]
[root@rhel6 ~]# chkconfig iptables off
4. 关闭节点上的 ACPI 服务:
[root@v1 ~]# /etc/init.d/acpid stop
正在关闭 acpi 守护进程: [确定]
[root@v1 ~]# chkconfig --level 2345 acpid off
[root@v1 ~]# chkconfig --list acpid
acpid 0:关闭 1:关闭 2:关闭 3:关闭 4:关闭 5:关闭 6:关闭
[root@v1 ~]#
5. 禁用 NetworkManager
[root@v1 ~]# /etc/init.d/NetworkManager stop
停止 NetworkManager 守护进程: [确定]
[root@v1 ~]# chkconfig NetworkManager off
6. 给软件用户 ricci 设置密码:
[root@v1 ~]# passwd ricci
更改用户 ricci 的密码 。
新的 密码:
重新输入新的 密码:
passwd: 所有的身份验证令牌已经成功更新。
[root@v1 ~]#
7. 启动 ricci 服务
[root@v1 ~]# /etc/init.d/ricci start
正在启动 oddjobd: [确定]
generating SSL certificates... done
Generating NSS database... done
启动 ricci: [确定]
[root@v1 ~]#
2.2.2. 集群配置管理器(luci)
可以安装在节点上,我现在安装在存储服务器上,更利于监控集群状态。
1. 配置好 yum 源后,安装管理软件:
[root@rhel6 ~]# yum install luci
2. 打开软件相应端口(iptables 命令)
端口
8084
协议
TCP
3. 或者直接禁用防火墙
程序
luci
[root@rhel6 ~]# /etc/init.d/iptables stop
iptables:清除防火墙规则: [确定]
iptables:将链设置为政策 ACCEPT:filter [确定]
iptables:正在卸载模块: [确定]
[root@rhel6 ~]# chkconfig iptables off
4. 启动 luci:
[root@rhel6 ~]# /etc/init.d/luci start
Adding following auto-detected host IDs (IP addresses/domain names),
corresponding to `rhel6.shenmin.com' address, to the configuration of
self-managed certificate `/var/lib/luci/etc/cacert.config' (you can change them
by editing `/var/lib/luci/etc/cacert.config', removing the generated certificate
`/var/lib/luci/certs/host.pem' and restarting luci):
(none suitable found, you can still do it manually as mentioned above)
Generating a 2048 bit RSA private key
writing new private key to '/var/lib/luci/certs/host.pem'
正在启动 saslauthd: [确定]
Start luci... [确定]
Point your web browser to https://rhel6.shenmin.com:8084 (or equivalent) to
access luci
[root@rhel6 ~]#
3. 创建集群
1. 登录管理服务器的 luci 界面:
2. 输入用户名 root 和密码,登录