在webhostingtalk上看到webnx 有一款机器在on sale，配置相当不错，而我又是webnx 的超级粉，立马拿下替换了原来旧的机器，新机器的配置如下:
Intel Xeon E3-1275v5 quadcore 3.6ghz 64gb ddr4 ECC ram 1TB Enterprise SSD /29 5x usable ip's 30Tb over 1gbps $65 ** limited time until we burn up the stock on hand!
实话实说，这个机器还是非常有新引力的，再加上webnx 100%的SLA 和 super connected network， 真的是非常爽的价格.
我准备用它来做ESXi 新的vCSA 主控.
在deploy ESXi之前，先来测试一下这台机器的硬件和network.
/dev/sda:
=== START OF INFORMATION SECTION === Model Family: Intel 730 and DC S35x0/3610/3700 Series SSDs Device Model: INTEL SSDSC2BA400G4 Serial Number: BTHV513606YB400NGN LU WWN Device Id: 5 5cd2e4 04b7c7e4a Firmware Version: G2010160 User Capacity: 400,088,457,216 bytes [400 GB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: Solid State Device Form Factor: 2.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Sep 18 22:16:03 2019 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled
可以看到SSD 是intel 的DC S3710
---------------------------------------------------------------------- CPU model : Intel(R) Xeon(R) CPU E3-1271 v3 @ 3.60GHz Number of cores : 8 CPU frequency : 1000.233 MHz Total size of Disk : 359.3 GB (1.1 GB Used) Total amount of Mem : 32129 MB (162 MB Used) Total amount of Swap : 7627 MB (0 MB Used) System uptime : 0 days, 21 hour 29 min Load average : 0.01, 0.03, 0.00 OS : Debian GNU/Linux 10 Arch : x86_64 (64 Bit) Kernel : 4.19.0-6-amd64 ---------------------------------------------------------------------- I/O speed(1st run) : 418 MB/s I/O speed(2nd run) : 418 MB/s I/O speed(3rd run) : 417 MB/s Average I/O speed : 417.7 MB/s ---------------------------------------------------------------------- Node Name IPv4 address Download Speed CacheFly 205.234.175.175 105MB/s Linode, Tokyo2, JP 139.162.65.37 18.7MB/s Linode, Singapore, SG 139.162.23.4 12.8MB/s Linode, London, UK 176.58.107.39 18.9MB/s Linode, Frankfurt, DE 139.162.130.8 17.0MB/s Linode, Fremont, CA 50.116.14.9 71.7MB/s Softlayer, Dallas, TX 173.192.68.18 30.1MB/s Softlayer, Seattle, WA 67.228.112.250 33.6MB/s Softlayer, Frankfurt, DE 159.122.69.4 8.31MB/s Softlayer, Singapore, SG 119.81.28.170 7.09MB/s Softlayer, HongKong, CN 119.81.130.170 7.78MB/s
第二遍测试:
---------------------------------------------------------------------- CPU model : Intel(R) Xeon(R) CPU E3-1271 v3 @ 3.60GHz Number of cores : 8 CPU frequency : 1277.243 MHz Total size of Disk : 359.3 GB (1.1 GB Used) Total amount of Mem : 32129 MB (164 MB Used) Total amount of Swap : 7627 MB (0 MB Used) System uptime : 0 days, 21 hour 32 min Load average : 0.07, 0.07, 0.01 OS : Debian GNU/Linux 10 Arch : x86_64 (64 Bit) Kernel : 4.19.0-6-amd64 ---------------------------------------------------------------------- I/O speed(1st run) : 419 MB/s I/O speed(2nd run) : 418 MB/s I/O speed(3rd run) : 418 MB/s Average I/O speed : 418.3 MB/s ---------------------------------------------------------------------- Node Name IPv4 address Download Speed CacheFly 205.234.175.175 105MB/s Linode, Tokyo2, JP 139.162.65.37 17.0MB/s Linode, Singapore, SG 139.162.23.4 13.0MB/s Linode, London, UK 176.58.107.39 19.6MB/s Linode, Frankfurt, DE 139.162.130.8 16.8MB/s Linode, Fremont, CA 50.116.14.9 85.3MB/s Softlayer, Dallas, TX 173.192.68.18 31.6MB/s Softlayer, Seattle, WA 67.228.112.250 32.7MB/s Softlayer, Frankfurt, DE 159.122.69.4 8.59MB/s Softlayer, Singapore, SG 119.81.28.170 7.08MB/s Softlayer, HongKong, CN 119.81.130.170 8.86MB/s ----------------------------------------------------------------------
/dev/sdb:
=== START OF INFORMATION SECTION === Model Family: Seagate Barracuda 3.5 Device Model: ST2000DM006-2DM164 Serial Number: Z560FAW9 LU WWN Device Id: 5 000c50 0a1e182e7 Firmware Version: CC26 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Sep 18 22:21:57 2019 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled
和online.net 一样，前面的测试有点不太规范，这次有机会重新测试一下
机器配置和online.net 一样， E5 1650V3， 64GB RAM， 2 X 1TB SSD ( micron 5100 enterprise SSD)
测试使用的程序是:
wget -qO- bench.sh | bash
直接测试IO，使用的是下面的commands:
dd if=/dev/zero of=test_$$ bs=64k count=16k conv=fdatasync && rm -f test_$$
直接在服务器裸机安装Centos 7，下面是测试结果:
[root@webnx ~]# hdparm -t /dev/sda /dev/sda: Timing buffered disk reads: 1422 MB in 3.00 seconds = 473.59 MB/sec [root@webnx ~]# hdparm -t /dev/sdb /dev/sdb: Timing buffered disk reads: 1452 MB in 3.00 seconds = 483.80 MB/sec [root@sda ~]# dd if=/dev/zero of=test_$$ bs=64k count=16k conv=fdatasync && rm -f test_$$ 16384+0 records in 16384+0 records out 1073741824 bytes (1.1 GB) copied, 2.75682 s, 389 MB/s
测试了几次，都在390 MB/s 左右
[root@sdb]# dd if=/dev/zero of=test_$$ bs=64k count=16k conv=fdatasync && rm -f test_$$ 16384+0 records in 16384+0 records out 1073741824 bytes (1.1 GB) copied, 2.75827 s, 389 MB/s 两块SSD 都是micron 的全新 1 TB SSD, 看来SSD 的速度就在400 MB/s 左右
用bench.sh的程序测试下:
---------------------------------------------------------------------- CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz Number of cores : 12 CPU frequency : 1199.920 MHz Total size of Disk : 851.3 GB (1.5 GB Used) Total amount of Mem : 64316 MB (564 MB Used) Total amount of Swap : 30516 MB (0 MB Used) System uptime : 0 days, 0 hour 8 min Load average : 0.08, 0.05, 0.03 OS : CentOS 7.7.1908 Arch : x86_64 (64 Bit) Kernel : 3.10.0-1062.4.1.el7.x86_64 ---------------------------------------------------------------------- I/O speed(1st run) : 391 MB/s I/O speed(2nd run) : 388 MB/s I/O speed(3rd run) : 389 MB/s Average I/O speed : 389.3 MB/s ----------------------------------------------------------------------
依然在390MB/s左右， micron 的SSD的表现要差于intel 和 samsung的SSD
下面看看在ESXi 6.7 上使用native drive 的表现:
Centos 7 with LVM
---------------------------------------------------------------------- CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz Number of cores : 4 CPU frequency : 3499.997 MHz Total size of Disk : 27.0 GB (1.3 GB Used) Total amount of Mem : 3950 MB (165 MB Used) Total amount of Swap : 3071 MB (0 MB Used) System uptime : 0 days, 0 hour 6 min Load average : 0.16, 0.10, 0.06 OS : CentOS 7.6.1810 Arch : x86_64 (64 Bit) Kernel : 3.10.0-957.el7.x86_64 ---------------------------------------------------------------------- I/O speed(1st run) : 469 MB/s I/O speed(2nd run) : 466 MB/s I/O speed(3rd run) : 468 MB/s Average I/O speed : 467.7 MB/s ----------------------------------
可以看到这个这个数据非常的好，基本接近原生SSD 的表现了
下面是Centos 7 without LVM
---------------------------------------------------------------------- CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz Number of cores : 4 CPU frequency : 3499.997 MHz Total size of Disk : 26.3 GB (2.3 GB Used) Total amount of Mem : 3950 MB (150 MB Used) Total amount of Swap : 3813 MB (0 MB Used) System uptime : 0 days, 0 hour 1 min Load average : 0.58, 0.23, 0.08 OS : CentOS 7.6.1810 Arch : x86_64 (64 Bit) Kernel : 3.10.0-957.el7.x86_64 ---------------------------------------------------------------------- I/O speed(1st run) : 458 MB/s I/O speed(2nd run) : 457 MB/s I/O speed(3rd run) : 460 MB/s Average I/O speed : 458.3 MB/s
Centos 7 with LVM and standard partition mounted on /sata,
---------------------------------------------------------------------- I/O speed(1st run) : 452 MB/s I/O speed(2nd run) : 456 MB/s I/O speed(3rd run) : 461 MB/s Average I/O speed : 456.3 MB/s ----------------------------------------------------------------------
/data上， SSD 的IO 也会降速，需要很长一段时间才能恢复满速
在Debian上测试一下(无论LVM还是standard partition结果都一样)：
---------------------------------------------------------------------- ---------------------------------------------------------------------- CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz Number of cores : 4 CPU frequency : 3499.997 MHz Total size of Disk : 94.2 GB (13.1 GB Used) Total amount of Mem : 4044 MB (129 MB Used) Total amount of Swap : 4191 MB (0 MB Used) System uptime : 0 days, 0 hour 9 min ---------------------------------------------------------------------- CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz Number of cores : 4 CPU frequency : 3499.997 MHz Total size of Disk : 94.2 GB (13.1 GB Used) Total amount of Mem : 4044 MB (129 MB Used) Total amount of Swap : 4191 MB (0 MB Used) System uptime : 0 days, 0 hour 15 min Load average : 0.02, 0.13, 0.14 OS : Debian GNU/Linux 10 Arch : x86_64 (64 Bit) Kernel : 4.19.0-6-amd64 ---------------------------------------------------------------------- I/O speed(1st run) : 456 MB/s I/O speed(2nd run) : 457 MB/s I/O speed(3rd run) : 462 MB/s Average I/O speed : 458.3 MB/s ---------------------------------------------------------------------- Node Name IPv4 address Download Speed CacheFly 205.234.175.175 109MB/s Linode, Tokyo2, JP 139.162.65.37 22.3MB/s Linode, Singapore, SG 139.162.23.4 Linode, London, UK 176.58.107.39 16.4MB/s Linode, Frankfurt, DE 139.162.130.8 15.1MB/s Linode, Fremont, CA 50.116.14.9 74.5MB/s Softlayer, Dallas, TX 173.192.68.18 51.5MB/s Softlayer, Seattle, WA 67.228.112.250 57.1MB/s Softlayer, Frankfurt, DE 159.122.69.4 14.0MB/s Softlayer, Singapore, SG 119.81.28.170 10.8MB/s Softlayer, HongKong, CN 119.81.130.170 12.7MB/s ---------------------------------------------------------------------- ----------------------------------------------------------------------
在Debian上，有的时候速度会降到200MB/s, 但是需要很长一段时间来恢复
下面来看看回退ssd 的driver 以后的表现:
回退driver以后，Debian 10 with LVM or without LVM
---------------------------------------------------------------------- CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz Number of cores : 4 CPU frequency : 3499.997 MHz Total size of Disk : 94.2 GB (13.1 GB Used) Total amount of Mem : 4044 MB (142 MB Used) Total amount of Swap : 4191 MB (0 MB Used) System uptime : 0 days, 12 hour 13 min Load average : 0.00, 0.00, 0.00 OS : Debian GNU/Linux 10 Arch : x86_64 (64 Bit) Kernel : 4.19.0-6-amd64 ---------------------------------------------------------------------- I/O speed(1st run) : 456 MB/s I/O speed(2nd run) : 461 MB/s I/O speed(3rd run) : 459 MB/s Average I/O speed : 458.7 MB/s
在测试中发现，SSD在standard partition上会降速，降速到200MB/s以后，会花费很长一段时间恢复到满速460MB/s, 但是在centos 7下，这个恢复是很快的
Centos 7 Standard Partition:
---------------------------------------------------------------------- I/O speed(1st run) : 455 MB/s I/O speed(2nd run) : 457 MB/s I/O speed(3rd run) : 457 MB/s Average I/O speed : 456.3 MB/s ----------------------------------------------------------------------
Centos 7 with LVM:
---------------------------------------------------------------------- I/O speed(1st run) : 464 MB/s I/O speed(2nd run) : 466 MB/s I/O speed(3rd run) : 462 MB/s Average I/O speed : 464.0 MB/s ----------------------------------------------------------------------
Centos 7 LVM with standard partition mounted on /data
---------------------------------------------------------------------- I/O speed(1st run) : 456 MB/s I/O speed(2nd run) : 461 MB/s I/O speed(3rd run) : 458 MB/s Average I/O speed : 458.3 MB/s ----------------------------------------------------------------------
在测试中发现，即使会有短暂的降速到200MB/s, 也会很快的恢复到460MB/s 的满速
通过以上的测试可以看到，vmware 全新的native driver，基本上解决了micron ssd 的low IO performance 的问题
micron 的ssd 偶尔会降速到200 MB/s，但是基本是可以恢复满速的，只是时间有长有短的问题。