在webhostingtalk上看到webnx 有一款机器在on sale，配置相当不错，而我又是webnx 的超级粉，立马拿下替换了原来旧的机器，新机器的配置如下:
Intel Xeon E3-1275v5 quadcore 3.6ghz 64gb ddr4 ECC ram 1TB Enterprise SSD /29 5x usable ip's 30Tb over 1gbps $65 ** limited time until we burn up the stock on hand!
实话实说，这个机器还是非常有新引力的，再加上webnx 100%的SLA 和 super connected network， 真的是非常爽的价格.
我准备用它来做ESXi 新的vCSA 主控.
在deploy ESXi之前，先来测试一下这台机器的硬件和network.
先检查硬盘信息:
[root@webnx ~]# smartctl -a /dev/sda smartctl 7.0 2018-12-30 r4883 [x86_64-linux-3.10.0-1062.1.2.el7.x86_64] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Samsung based SSDs Device Model: Samsung SSD 860 EVO 1TB Serial Number: S3Z6NB0M106674P LU WWN Device Id: 5 002538 e40c2fc1b Firmware Version: RVT02B6Q User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Form Factor: 2.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Thu Oct 17 23:51:22 2019 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled
1TB 的SSD 给的是samsung 的 860 EVO， 也算是enterprise 的SSD，通电时间只有815 hours，非常的爽
[root@webnx ~]# smartctl -a /dev/sdb smartctl 7.0 2018-12-30 r4883 [x86_64-linux-3.10.0-1062.1.2.el7.x86_64] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST1000DM003-1CH162 Serial Number: Z1D7JRQN LU WWN Device Id: 5 000c50 05d0bd293 Firmware Version: CC49 User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Thu Oct 17 23:54:01 2019 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled
1TB 的备份盘给的是希捷的barracuda盘，通电时间是1435 hours，也是非常不错的
下面主要测试一下IO， 测试使用的程序是
wget -qO- bench.sh | bash
下面是测试结果:
---------------------------------------------------------------------- CPU model : Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz Number of cores : 8 CPU frequency : 877.368 MHz Total size of Disk : 888.3 GB (1.5 GB Used) Total amount of Mem : 64252 MB (521 MB Used) Total amount of Swap : 30516 MB (0 MB Used) System uptime : 0 days, 1 hour 4 min Load average : 0.07, 0.03, 0.11 OS : CentOS 7.7.1908 Arch : x86_64 (64 Bit) Kernel : 3.10.0-1062.1.2.el7.x86_64 ---------------------------------------------------------------------- I/O speed(1st run) : 435 MB/s I/O speed(2nd run) : 437 MB/s I/O speed(3rd run) : 436 MB/s Average I/O speed : 436.0 MB/s ---------------------------------------------------------------------- Node Name IPv4 address Download Speed CacheFly 205.234.175.175 105MB/s Linode, Tokyo2, JP 139.162.65.37 17.6MB/s Linode, Singapore, SG 139.162.23.4 12.5MB/s Linode, London, UK 176.58.107.39 18.6MB/s Linode, Frankfurt, DE 139.162.130.8 16.8MB/s Linode, Fremont, CA 50.116.14.9 89.1MB/s Softlayer, Dallas, TX 173.192.68.18 8.54MB/s Softlayer, Seattle, WA 67.228.112.250 28.9MB/s Softlayer, Frankfurt, DE 159.122.69.4 7.12MB/s Softlayer, Singapore, SG 119.81.28.170 6.79MB/s Softlayer, HongKong, CN 119.81.130.170 7.89MB/s
测试了即便，860 evo 的IO 速度一直保持在436MB/s, 还是不错的
下面是测试sata 硬盘:
I/O speed(1st run) : 152 MB/s I/O speed(2nd run) : 148 MB/s I/O speed(3rd run) : 154 MB/s Average I/O speed : 151.3 MB/s
测试了几次，发现IO speed基本都维持在153MB/s 左右， 这个IO 也确实不错的说
再用hdparm测试一下IO
[root@webnx ~]# hdparm -t /dev/sda /dev/sda: Timing buffered disk reads: 1532 MB in 3.00 seconds = 510.02 MB/sec [root@webnx ~]# hdparm -t /dev/sdb /dev/sdb: Timing buffered disk reads: 430 MB in 3.01 seconds = 143.06 MB/sec
和用dd 测试IO 速度差别不大
下面开始ESXi VM IO测试:
- CentOS 7 使用新的ACHI Driver: vmw-achi,使用LVM
---------------------------------------------------------------------- CPU model : Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz Number of cores : 4 CPU frequency : 3600.000 MHz Total size of Disk : 46.0 GB (1.4 GB Used) Total amount of Mem : 4048 MB (208 MB Used) Total amount of Swap : 4095 MB (0 MB Used) System uptime : 0 days, 0 hour 2 min Load average : 0.35, 0.09, 0.03 OS : CentOS 7.7.1908 Arch : x86_64 (64 Bit) Kernel : 3.10.0-1062.el7.x86_64 ---------------------------------------------------------------------- I/O speed(1st run) : 499 MB/s I/O speed(2nd run) : 499 MB/s I/O speed(3rd run) : 498 MB/s Average I/O speed : 498.7 MB/s ---------------------------------------------------------------------- Node Name IPv4 address Download Speed CacheFly 205.234.175.175 105MB/s Linode, Tokyo2, JP 139.162.65.37 18.5MB/s Linode, Singapore, SG 139.162.23.4 12.8MB/s Linode, London, UK 176.58.107.39 19.1MB/s Linode, Frankfurt, DE 139.162.130.8 16.9MB/s Linode, Fremont, CA 50.116.14.9 89.4MB/s Softlayer, Dallas, TX 173.192.68.18 43.5MB/s Softlayer, Seattle, WA 67.228.112.250 43.6MB/s Softlayer, Frankfurt, DE 159.122.69.4 9.48MB/s Softlayer, Singapore, SG 119.81.28.170 8.36MB/s Softlayer, HongKong, CN 119.81.130.170 10.3MB/s
2. CentOS 7 使用新的ACHI Driver: vmw-achi, 不使用LVM
在ssd上：
---------------------------------------------------------------------- CPU model : Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz Number of cores : 4 CPU frequency : 3600.000 MHz Total size of Disk : 96.3 GB (1.5 GB Used) Total amount of Mem : 4048 MB (207 MB Used) Total amount of Swap : 3813 MB (0 MB Used) System uptime : 0 days, 0 hour 4 min Load average : 0.50, 0.18, 0.07 OS : CentOS 7.7.1908 Arch : x86_64 (64 Bit) Kernel : 3.10.0-1062.el7.x86_64 ---------------------------------------------------------------------- I/O speed(1st run) : 498 MB/s I/O speed(2nd run) : 499 MB/s I/O speed(3rd run) : 499 MB/s Average I/O speed : 498.7 MB/s ---------------------------------------------------------------------- Node Name IPv4 address Download Speed CacheFly 205.234.175.175 105MB/s Linode, Tokyo2, JP 139.162.65.37 17.9MB/s Linode, Singapore, SG 139.162.23.4 12.9MB/s Linode, London, UK 176.58.107.39 17.9MB/s Linode, Frankfurt, DE 139.162.130.8 16.5MB/s Linode, Fremont, CA 50.116.14.9 89.7MB/s Softlayer, Dallas, TX 173.192.68.18 42.6MB/s Softlayer, Seattle, WA 67.228.112.250 48.8MB/s Softlayer, Frankfurt, DE 159.122.69.4 9.74MB/s Softlayer, Singapore, SG 119.81.28.170 8.83MB/s Softlayer, HongKong, CN 119.81.130.170 10.5MB/s
在sata上, 使用LVM:
---------------------------------------------------------------------- I/O speed(1st run) : 148 MB/s I/O speed(2nd run) : 149 MB/s I/O speed(3rd run) : 148 MB/s Average I/O speed : 148.3 MB/s ----------------------------------------------------------------------
在sata上不使用LVM:
---------------------------------------------------------------------- I/O speed(1st run) : 142 MB/s I/O speed(2nd run) : 146 MB/s I/O speed(3rd run) : 145 MB/s Average I/O speed : 144.3 MB/s ----------------------------------------------------------------------
由此可见， ESXi 在6.7 u3中应该修复了新的driver 的问题，在VM 中的IO测试和在裸机上的IO测试基本差不多。
下面我们来看看回退在旧的driver 的情况下， IO 的表现如何.
Centos7 with LVM on SSD:
---------------------------------------------------------------------- I/O speed(1st run) : 498 MB/s I/O speed(2nd run) : 499 MB/s I/O speed(3rd run) : 501 MB/s Average I/O speed : 499.3 MB/s ----------------------------------------------------------------------
Centos7 Without LVM on SSD:
---------------------------------------------------------------------- I/O speed(1st run) : 500 MB/s I/O speed(2nd run) : 499 MB/s I/O speed(3rd run) : 499 MB/s Average I/O speed : 499.3 MB/s ----------------------------------------------------------------------
Centos7 without LVM on SSD mounted on /data
---------------------------------------------------------------------- I/O speed(1st run) : 488 MB/s I/O speed(2nd run) : 489 MB/s I/O speed(3rd run) : 489 MB/s Average I/O speed : 488.7 MB/s ----------------------------------------------------------------------
Centos7 with LVM on SATA
---------------------------------------------------------------------- I/O speed(1st run) : 147 MB/s I/O speed(2nd run) : 148 MB/s I/O speed(3rd run) : 147 MB/s Average I/O speed : 147.3 MB/s ----------------------------------------------------------------------
Centos7 witout LVM on SATA mounted on /data
---------------------------------------------------------------------- I/O speed(1st run) : 139 MB/s I/O speed(2nd run) : 139 MB/s I/O speed(3rd run) : 136 MB/s Average I/O speed : 138.0 MB/s ----------------------------------------------------------------------
最后总结，在ESXi 6.7 u3这个版本上，新的sata 驱动vmw_ahci已经能够很好的驱动各种ssd 和 sata了，而且表现还很好，不需要再去回退到旧的驱动了