Online.net Pro-4-M 独立服务器IO Benchmark

很是喜欢online.net 家的服务,以为他们家都是使用的Dell 的服务器。

今天准备再增加一台服务器专门来管理服务器集群的时候,才发现online只有在中高端服务器上采用Dell EMC 和 Dell 的服务器,在中低端市场上已online labs 自己的服务器以及QTS 服务器为主,中端市场也有一部分是Supermicro 服务器。

因为主要是拿这台服务器做管理以及存放一些静态ISO镜像,所以需求的配置不要求高,300 GB 加 500GB SSD 就可以了,最好还是SuperMicro 主板,选来选去就只有Pro-4-M 符合条件了。OVH 倒是也有一些符合条件的,就是经过验证的OVH账号实在太难获得了.

和Pro-4-L 不同,这台机器带的SSD是三星的,型号是PM871b,用国内的话来说,是850 evo 的马甲。看来online 的服务器用家用级别的SSD 多一点

测试使用的程序:

wget -qO- bench.sh | bash

首先是在裸机上安装CentOS 测试.

----------------------------------------------------------------------
CPU model : Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz
Number of cores : 8
CPU frequency : 1099.945 MHz
Total size of Disk : 461.3 GB (1.5 GB Used)
Total amount of Mem : 64050 MB (485 MB Used)
Total amount of Swap : 8511 MB (0 MB Used)
System uptime : 0 days, 1 hour 24 min
Load average : 0.00, 0.01, 0.05
OS : CentOS 7.6.1810
Arch : x86_64 (64 Bit)
Kernel : 3.10.0-957.27.2.el7.x86_64
----------------------------------------------------------------------
I/O speed(1st run) : 427 MB/s
I/O speed(2nd run) : 427 MB/s
I/O speed(3rd run) : 427 MB/s
Average I/O speed : 427.0 MB/s

 

[root@sd ~]# hdparm -t /dev/sda

/dev/sda:
Timing buffered disk reads: 1554 MB in 3.00 seconds = 517.37 MB/sec
[root@sd ~]# hdparm -t /dev/sdb

/dev/sdb:
Timing buffered disk reads: 1356 MB in 3.00 seconds = 451.67 MB/sec
[root@sd ~]# hdparm -t /dev/sdc

/dev/sdc:
Timing buffered disk reads: 1348 MB in 3.00 seconds = 449.08 MB/sec

服务器安装系统常用信息和链接

虚拟光驱:

https://www.elby.ch/en/products/vcd.html

1) NTP 服务器:

https://www.ntppool.org/en/use.html

   常用服务器:

0.pool.ntp.org
1.pool.ntp.org
2.pool.ntp.org
3.pool.ntp.org

2) CentOS 下载:

镜像地址:

https://www.centos.org/download/mirrors/

根据IP直接展示最近的镜像地址:

http://isoredirect.centos.org/centos/7/isos/x86_64/

FR:

http://centos.mirrors.proxad.net/7.6.1810/isos/x86_64/CentOS-7-x86_64-NetInstall-1810.iso

Centos 7 Netinstall URL:

http://mirror.centos.org/centos/7/os/x86_64/
http://mirror.xtom.com.hk/centos/    (Hong Kong)
http://mirror.nus.edu.sg/centos/ (Singapore)
http://centos.mirror.netelligent.ca/centos/   (Canada)
http://centos.mirrors.proxad.net/  (France)
http://mirror.fra10.de.leaseweb.net/centos/ (Germany)
http://mirror.ams1.nl.leaseweb.net/centos/  (Netherlands)
http://centos.mirror.transip.nl/  (Netherlands)
http://repos.lax.quadranet.com/centos/ (US - QuadraNet)
http://mirror.facebook.net/centos/ (US - Facebook)
http://mirror.sfo12.us.leaseweb.net/centos/ (US - Leaseweb)
http://mirrors.gigenet.com/centos/  (US - Gigenet)
http://mirror.steadfastnet.com/centos/ (US - Steadfast)

 

CentOS 7 Sha256 校验:

3213b2c34cecbb3bb817030c7f025396b658634c0cf9c4435fc0b52ec9644667 CentOS-7-x86_64-LiveGNOME-1810.iso
38d5d51d9d100fd73df031ffd6bd8b1297ce24660dc8c13a3b8b4534a4bd291c CentOS-7-x86_64-Minimal-1810.iso
6d44331cc4f6c506c7bbe9feb8468fad6c51a88ca1393ca6b8b486ea04bec3c1 CentOS-7-x86_64-DVD-1810.iso
87623c8ab590ad0866c5f5d86a2d7ed631c61d69f38acc42ce2c8ddec65ecea2 CentOS-7-x86_64-LiveKDE-1810.iso
918975cdf947e858c9a0c77d6b90a9a56d9977f3a4496a56437f46f46200cf71 CentOS-7-x86_64-Everything-1810.iso
19d94274ef856c4dfcacb2e7cfe4be73e442a71dd65cc3fb6e46db826040b56e CentOS-7-x86_64-NetInstall-1810.iso

 

3) Debian 最新版下载:

https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.0.0-amd64-netinst.iso

Sha256 校验:

3dbb597b7f11dbda71cda08d4c1339c1eb565e784c75409987fa2b91182d9240 debian-10.0.0-amd64-netinst.iso
5714b75bbf02c61c2eacad51b6042efd0806fcc60180ad9029f15cace21119fb debian-10.0.0-amd64-xfce-CD-1.iso
58405101948aff46d5261e0a4d3e8f98ea3dc5366a9ada881ebe921eafd5f737 debian-edu-10.0.0-amd64-netinst.iso
de14801ab29ea71e606ff516e87b1ca68499de21098f994778f77121ea41323d debian-mac-10.0.0-amd64-netinst.iso

 

Debian 9.9 下载:

https://cdimage.debian.org/cdimage/archive/9.9.0/amd64/iso-cd/debian-9.9.0-amd64-netinst.iso

Sha256 校验:

d4a22c81c76a66558fb92e690ef70a5d67c685a08216701b15746586520f6e8e debian-9.9.0-amd64-netinst.iso
6fa24e8e305bdfd762605f092c2dcddb2e76f8522cb9a5035df88532d28c9a50 debian-9.9.0-amd64-xfce-CD-1.iso
4de26144286993ac6f4f244ff332fdd80e11ec230906e7997aa426e775fac1fb debian-mac-9.9.0-amd64-netinst.iso

 

vCSA Hardware Requirements

在ESXi 6.7 上,使用vCSA 的话, tiny environment 是足够的. 但是在最新版本的vCSA 6.7中,总是会出现内存不足的warning。

经过一番google 知道了,VMWARE 在vCSA中主要使用的是Java,这是Java 的内存机制造成的,简单的解决办法就是直接增加vCSA 的内存. 在 ESXi 6.0 以后, vCSA 的内存是可以在VM 上直接更改的,这是很方便的事情。

另外如果服务器硬件足够,建议直接deploy 一个small 或者 medium environment 来直接解决这个问题。

下图是vCSA 的硬件需求,以后就不用每次都去翻官网了.

 

 

Small environment 都需要4 vCPU, 16 GB 的RAM,storage 也需要340GB, 最好是SSD

这个需求,说实话,真的比很多德利服务器的配置都高了。

不过幸好online.net, OVH 以及hetzner 这样的服务商的存在。。

推荐online.net 和OVH, 起码都是企业级的硬件和网络, hetzner 就要差点了

WebNX.com 服务器重新benchmark

和online.net 一样,前面的测试有点不太规范,这次有机会重新测试一下

机器配置和online.net 一样, E5 1650V3, 64GB RAM, 2 X 1TB SSD ( micron 5100 enterprise SSD)

测试使用的程序是:

wget -qO- bench.sh | bash

直接测试IO,使用的是下面的commands:

dd if=/dev/zero of=test_$$ bs=64k count=16k conv=fdatasync && rm -f test_$$

直接在服务器裸机安装Centos 7,下面是测试结果:

[root@webnx ~]# hdparm -t /dev/sda

/dev/sda:
Timing buffered disk reads: 1422 MB in 3.00 seconds = 473.59 MB/sec
[root@webnx ~]# hdparm -t /dev/sdb

/dev/sdb:
Timing buffered disk reads: 1452 MB in 3.00 seconds = 483.80 MB/sec

[root@sda ~]# dd if=/dev/zero of=test_$$ bs=64k count=16k conv=fdatasync && rm -f test_$$
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 2.75682 s, 389 MB/s

测试了几次,都在390 MB/s 左右

[root@sdb]# dd if=/dev/zero of=test_$$ bs=64k count=16k conv=fdatasync && rm -f test_$$
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 2.75827 s, 389 MB/s

两块SSD 都是micron 的全新 1 TB SSD, 看来SSD 的速度就在400 MB/s 左右

下面看看在ESXi 6.7 上使用native drive 的表现:

Centos 7 with LVM

----------------------------------------------------------------------
CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
Number of cores : 4
CPU frequency : 3499.997 MHz
Total size of Disk : 27.0 GB (1.3 GB Used)
Total amount of Mem : 3950 MB (165 MB Used)
Total amount of Swap : 3071 MB (0 MB Used)
System uptime : 0 days, 0 hour 6 min
Load average : 0.16, 0.10, 0.06
OS : CentOS 7.6.1810
Arch : x86_64 (64 Bit)
Kernel : 3.10.0-957.el7.x86_64
----------------------------------------------------------------------
I/O speed(1st run) : 469 MB/s
I/O speed(2nd run) : 466 MB/s
I/O speed(3rd run) : 468 MB/s
Average I/O speed : 467.7 MB/s
----------------------------------

可以看到这个这个数据非常的好,基本接近原生SSD 的表现了

下面是Centos 7 without LVM

----------------------------------------------------------------------
CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
Number of cores : 4
CPU frequency : 3499.997 MHz
Total size of Disk : 26.3 GB (2.3 GB Used)
Total amount of Mem : 3950 MB (150 MB Used)
Total amount of Swap : 3813 MB (0 MB Used)
System uptime : 0 days, 0 hour 1 min
Load average : 0.58, 0.23, 0.08
OS : CentOS 7.6.1810
Arch : x86_64 (64 Bit)
Kernel : 3.10.0-957.el7.x86_64
----------------------------------------------------------------------
I/O speed(1st run) : 176 MB/s
I/O speed(2nd run) : 182 MB/s
I/O speed(3rd run) : 178 MB/s
Average I/O speed : 178.7 MB/s

 

下面来看看回退ssd 的driver 以后的表现:

----------------------------------------------------------------------
CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
Number of cores : 4
CPU frequency : 3499.997 MHz
Total size of Disk : 26.3 GB (2.3 GB Used)
Total amount of Mem : 3950 MB (137 MB Used)
Total amount of Swap : 3813 MB (0 MB Used)
System uptime : 0 days, 0 hour 0 min
Load average : 0.12, 0.03, 0.01
OS : CentOS 7.6.1810
Arch : x86_64 (64 Bit)
Kernel : 3.10.0-957.el7.x86_64
----------------------------------------------------------------------
I/O speed(1st run) : 455 MB/s
I/O speed(2nd run) : 458 MB/s
I/O speed(3rd run) : 458 MB/s
Average I/O speed : 457.0 MB/s

回退driver以后,Centos 7 with LVM

----------------------------------------------------------------------
CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
Number of cores : 4
CPU frequency : 3499.997 MHz
Total size of Disk : 37.0 GB (1.2 GB Used)
Total amount of Mem : 3950 MB (159 MB Used)
Total amount of Swap : 3967 MB (0 MB Used)
System uptime : 0 days, 0 hour 3 min
Load average : 0.34, 0.11, 0.04
OS : CentOS 7.6.1810
Arch : x86_64 (64 Bit)
Kernel : 3.10.0-957.el7.x86_64
----------------------------------------------------------------------
I/O speed(1st run) : 463 MB/s
I/O speed(2nd run) : 466 MB/s
I/O speed(3rd run) : 461 MB/s
Average I/O speed : 463.3 MB/s

Online.net Pro-4-L 重新benchmark

前面的测试有点不太规范,这次又机会重新测试一下。

机器配置是Xeon E5 1650V3, 96GB RAM, 3X500GB SSD

测试使用的程序是:

wget -qO- bench.sh | bash

直接测试IO,使用的是下面的commands:

dd if=/dev/zero of=test_$$ bs=64k count=16k conv=fdatasync && rm -f test_$$

直接在服务器裸机上安装Centos 7, 下面是测试结果:

[root@sd ~]# hdparm -t /dev/sda

/dev/sda:
Timing buffered disk reads: 1454 MB in 3.00 seconds = 484.53 MB/sec

[root@sd ~]# hdparm -t /dev/sdb

/dev/sdb:
Timing buffered disk reads: 1472 MB in 3.00 seconds = 490.04 MB/sec

[root@sd ~]# hdparm -t /dev/sdc

/dev/sdc:
iming buffered disk reads: 1470 MB in 3.00 seconds = 489.96 MB/sec
[root@sd sda~]# dd if=/dev/zero of=test_$$ bs=64k count=16k conv=fdatasync && rm -f test_$$
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 2.65959 s, 404 MB/s

测试了几次,都在400 MB/s 左右

[root@sd sdb]# dd if=/dev/zero of=test_$$ bs=64k count=16k conv=fdatasync && rm -f test_$$
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 2.69217 s, 399 MB/s
[root@sd sdc]# dd if=/dev/zero of=test_$$ bs=64k count=16k conv=fdatasync && rm -f test_$$
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 2.70218 s, 397 MB/s

可以看到裸机直接测试的话,SSD 的IO 一般就在400 MB/s 左右

在Host上安装上Esxi 6.7 以后,分别测试Debian 9 和 Debian 10

Debian 9 测试结果:

----------------------------------------------------------------------
CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
Number of cores : 8
CPU frequency : 3495.142 MHz
Total size of Disk : 55.0 GB (0.9 GB Used)
Total amount of Mem : 3907 MB (92 MB Used)
Total amount of Swap : 4045 MB (0 MB Used)
System uptime : 0 days, 0 hour 11 min
Load average : 0.96, 0.28, 0.09
OS : Debian GNU/Linux 9
Arch : x86_64 (64 Bit)
Kernel : 4.9.0-9-amd64
----------------------------------------------------------------------
I/O speed(1st run) : 82.2 MB/s
I/O speed(2nd run) : 79.9 MB/s
I/O speed(3rd run) : 80.1 MB/s
Average I/O speed : 80.7 MB/s

Debian 10 测试结果:

----------------------------------------------------------------------
CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
Number of cores : 8
CPU frequency : 3495.951 MHz
Total size of Disk : 55.0 GB (1.0 GB Used)
Total amount of Mem : 3897 MB (150 MB Used)
Total amount of Swap : 4045 MB (0 MB Used)
System uptime : 0 days, 0 hour 25 min
Load average : 1.14, 0.31, 0.11
OS : Debian GNU/Linux 10
Arch : x86_64 (64 Bit)
Kernel : 4.19.0-5-amd64
----------------------------------------------------------------------
I/O speed(1st run) : 78.0 MB/s
I/O speed(2nd run) : 77.0 MB/s
I/O speed(3rd run) : 76.9 MB/s
Average I/O speed : 77.3 MB/s

本来以为Debian 10 有什么特殊科技呢,IO 测试和Debian 9 差不多

下面是重头戏,回退一些Esxi 的 ACHI 的driver,再来测试一下:

----------------------------------------------------------------------
CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
Number of cores : 8
CPU frequency : 3494.956 MHz
Total size of Disk : 55.0 GB (0.9 GB Used)
Total amount of Mem : 3907 MB (91 MB Used)
Total amount of Swap : 4045 MB (0 MB Used)
System uptime : 0 days, 0 hour 4 min
Load average : 0.22, 0.05, 0.02
OS : Debian GNU/Linux 9
Arch : x86_64 (64 Bit)
Kernel : 4.9.0-9-amd64
----------------------------------------------------------------------
I/O speed(1st run) : 301 MB/s
I/O speed(2nd run) : 299 MB/s
I/O speed(3rd run) : 303 MB/s
Average I/O speed : 301.0 MB/s

数据在350-370 之间乱跳

在Debian 10上,闲时IO 能到400 MB/s, 平时也基本就是300-350左右

----------------------------------------------------------------------
CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
Number of cores : 8
CPU frequency : 3495.951 MHz
Total size of Disk : 55.0 GB (2.0 GB Used)
Total amount of Mem : 3897 MB (148 MB Used)
Total amount of Swap : 4045 MB (0 MB Used)
System uptime : 0 days, 0 hour 21 min
Load average : 0.00, 0.01, 0.01
OS : Debian GNU/Linux 10
Arch : x86_64 (64 Bit)
Kernel : 4.19.0-5-amd64
----------------------------------------------------------------------
I/O speed(1st run) : 306 MB/s
I/O speed(2nd run) : 300 MB/s
I/O speed(3rd run) : 299 MB/s
Average I/O speed : 301.7 MB/s

Vultr VM Benchmark

测试使用程序为:

wget -qO- bench.sh | bash

下面是测试结果:

----------------------------------------------------------------------
CPU model : Virtual CPU 82d9ed4018dd
Number of cores : 1
CPU frequency : 2593.912 MHz
Total size of Disk : 55.0 GB (0.9 GB Used)
Total amount of Mem : 2004 MB (53 MB Used)
Total amount of Swap : 0 MB (0 MB Used)
System uptime : 0 days, 0 hour 3 min
Load average : 0.31, 0.18, 0.07
OS : Debian GNU/Linux 9
Arch : x86_64 (64 Bit)
Kernel : 4.9.0-9-amd64
----------------------------------------------------------------------
I/O speed(1st run) : 474 MB/s
I/O speed(2nd run) : 475 MB/s
I/O speed(3rd run) : 470 MB/s
Average I/O speed : 473.0 MB/s

WebNX 1TB SSD IO Benchmark

Quadranet 的 硬件实在太老了,WebNX 以硬件好,网络好而著称, 刚订购的服务器自带两个1TB SSD,而且通过smart 查询到,是全新的SSD,真是太爽了

测试使用的程序:

wget -qO- bench.sh | bash

先在promox 6.0 的Host 上测试一下IO:

----------------------------------------------------------------------
CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
Number of cores : 12
CPU frequency : 3600.230 MHz
Total size of Disk : 94.0 GB (1.7 GB Used)
Total amount of Mem : 64307 MB (1237 MB Used)
Total amount of Swap : 8191 MB (0 MB Used)
System uptime : 0 days, 0 hour 34 min
Load average : 0.00, 0.00, 0.00
OS : Debian GNU/Linux 10
Arch : x86_64 (64 Bit)
Kernel : 5.0.15-1-pve
----------------------------------------------------------------------
I/O speed(1st run) : 406 MB/s
I/O speed(2nd run) : 406 MB/s
I/O speed(3rd run) : 405 MB/s
Average I/O speed : 405.7 MB/s

然后再proxmox上建立一个CentOS 7 的VM,在测试一下IO:

----------------------------------------------------------------------
CPU model : Common KVM processor
Number of cores : 4
CPU frequency : 3499.996 MHz
Total size of Disk : 29.0 GB (1.2 GB Used)
Total amount of Mem : 4358 MB (124 MB Used)
Total amount of Swap : 3275 MB (0 MB Used)
System uptime : 0 days, 0 hour 2 min
Load average : 0.20, 0.05, 0.02
OS : CentOS 7.6.1810
Arch : x86_64 (64 Bit)
Kernel : 3.10.0-957.el7.x86_64
----------------------------------------------------------------------
I/O speed(1st run) : 452 MB/s
I/O speed(2nd run) : 455 MB/s
I/O speed(3rd run) : 448 MB/s
Average I/O speed : 451.7 MB/s

下面在裸机上直接安装CentOS,来测试一下IO Benchmark:

----------------------------------------------------------------------
CPU model : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
Number of cores : 12
CPU frequency : 1199.920 MHz
Total size of Disk : 873.3 GB (1.3 GB Used)
Total amount of Mem : 64316 MB (503 MB Used)
Total amount of Swap : 7628 MB (0 MB Used)
System uptime : 0 days, 0 hour 7 min
Load average : 0.13, 0.07, 0.05
OS : CentOS 7.6.1810
Arch : x86_64 (64 Bit)
Kernel : 3.10.0-957.el7.x86_64
----------------------------------------------------------------------
I/O speed(1st run) : 390 MB/s
I/O speed(2nd run) : 390 MB/s
I/O speed(3rd run) : 389 MB/s
Average I/O speed : 389.7 MB/s

看来Proxmox 是有什么cache加成,造成IO 的测试最快