Mysql/Mariadb 优化

我不擅长数据库优化,但是今天在stackoverflow上看到有perl脚本来提供优化的suggestion,真心不错

这个脚本的名字叫MySqlTuber

The MySQLTuner script assesses your MySQL installation, and then outputs suggestions for increasing your server’s performance and stability.

Download the MySQLTuner script:

wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/mysqltuner.pl

Change the scripts permissions to be executable:

chmod +x mysqltuner.pl

Run the mysqltuner.pl script. You will be prompted to enter in your MySQL administrative login and password:

./mysqltuner.pl

The script will return results similar to the output below:

>> MySQLTuner 1.4.0 - Major Hayden <[email protected]>
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with '--help' for additional options and output filtering
Please enter your MySQL administrative login: root
Please enter your MySQL administrative password:
[OK] Currently running supported MySQL version 5.5.41-0+wheezy1
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MRG_MYISAM
[--] Data in InnoDB tables: 1M (Tables: 11)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[!!] Total fragmented tables: 11

-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 47s (113 q [2.404 qps], 42 conn, TX: 19K, RX: 7K)
[--] Reads / Writes: 100% / 0%
[--] Total buffers: 192.0M global + 2.7M per thread (151 max threads)
[OK] Maximum possible memory usage: 597.8M (60% of installed RAM)
[OK] Slow queries: 0% (0/113)
[OK] Highest usage of available connections: 0% (1/151)
[OK] Key buffer size / total MyISAM indexes: 16.0M/99.0K
[!!] Query cache efficiency: 0.0% (0 cached / 71 selects)
[OK] Query cache prunes per day: 0
[OK] Temporary tables created on disk: 25% (54 on disk / 213 total)
[OK] Thread cache hit rate: 97% (1 created / 42 connections)
[OK] Table cache hit rate: 24% (52 open / 215 opened)
[OK] Open file limit used: 4% (48/1K)
[OK] Table locks acquired immediately: 100% (62 immediate / 62 locks)
[OK] InnoDB buffer pool / data size: 128.0M/1.2M
[OK] InnoDB log waits: 0
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
Enable the slow query log to troubleshoot bad queries
Variables to adjust:
query_cache_limit (> 1M, or use smaller result sets)

MySQLTuner offers suggestions regarding how to better the database’s performance. If you are wary about updating your database on your own, following MySQLTuner’s suggestions is one of the safer ways to improve your database performance.

Mariadb 无故宕机

最近有一台VM上的mariadb总是无故crash,在stackoverflow上搜了一下,发现了一些简单的debug方法。

其实对于大部分的mariadb无故crash来说,基本都是内存满了.

1 先检查mariadb状态, 这个不用说大家都会

systemctl status mariadb

2 如果出现了SIGKILL,那基本上都是OOM Killer干掉了, 那你就可以通过如下命令来查看是哪个process被干掉了

dmesg -T | egrep -i 'killed process'

也可以通过下面的命令来查看全部log

dmesg -T | less

 

Stackoverflow上的其他辅助回答:

The first is to look at your configuration for your webserver and the database (and presumably there some application tier in here as well) to ensure that they don’t try to use up more RAM than you have available. Since you didn’t tell us about the application server nor the webserver I can’t really advise (but here’s a hint for pre-fork Apache, optionally with PHP). As for Mariadb – go get a copy of mysqltuner.pl and run it against your installation.

That should prevent you running out of memory most of the time, but you should also reduce the amount of non-existent memory the kernel will hand out:

sysctl vm.overcommit_memory=2
sysctl vm.overcommit_ratio=20

and try lower values for the ratio if you are still seeing OOM killer.

 

the row size is xx which is greater than maximum allowed size (8126 bytes) for a record on index leaf page

一个数据库DDL在MariaDB 10.3上没有任何问题,在MariaDB 10.6上就一般报这个问题:

[ERROR] InnoDB: Cannot add field `footer_color` in table `xxxx`.`theme` because after adding it, the row size is 8474 which is greater than maximum allowed size (8126 bytes) for a record on index leaf page.

这个Google一下很容易就知道是Row Size Too Large Error. 根据下面的文档很容易解决办法.

https://mariadb.com/kb/en/troubleshooting-row-size-too-large-errors-with-innodb/

但是按理说MariaDB 10.3 和 MariaDB 10.6上在这方面变动不大,为什么会出现问题呢?

搜索了半天,在MariaDB的文档上找到了说明:

https://mariadb.com/kb/en/innodb-row-formats-overview/#maximum-row-size

原来在MariaDB 10.2.26, 10.3.17, 10.4.7之前的版本上,InnoDB计算row size的方式是错误的,在后面的版本里修复了。

而我的MariaDB 10.3的版本正好是10.3.15.

困扰了一个上午的问题解决了!

另附解决办法,如果不想修改表的结构,可以在my.cnf添加下面的命令:

innodb_strict_mode = OFF

阿里云ECS的坑 – 系统硬件时间

用了这么久 的阿里云ecs,也是今天无意间才发现的,更准确的说应该是centos 7的坑,在centos 6上那样直接修改时区的办法不行了

1)拿到了阿里云的ecs,第一件事就是应该修改系统的hwclock,这是因为阿里云的ecs默认使用的是hwclock 是cst,也就是中国时区。这个可以是可以,但是国际上通用的办法是使用UTC时间,这更能避免很多问题:

timedatectl set-local-rtc 0

2) 第二个坑就是因为有了第一个问题以后,在centos 6上我经常使用下面的办法修改时区(个人习惯,所有的服务器都是UTC或者EST时区):

cp /usr/share/zoneinfo/America/New_York /etc/localtime

在centos 7上我也继续这么使用,在国外的服务器上没有一点问题,但是在阿里云ecs上就会出现下面的情况:

[root@sz ~]# timedatectl
Local time: Wed 2020-01-29 07:57:08 EST
Universal time: Wed 2020-01-29 12:57:08 UTC
RTC time: Wed 2020-01-29 20:57:07
Time zone: Asia/Shanghai (EST, -0500)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: yes
DST active: no
Last DST change: DST ended at
Sun 2019-11-03 01:59:59 EDT
Sun 2019-11-03 01:00:00 EST
Next DST change: DST begins (the clock jumps one hour forward) at
Sun 2020-03-08 01:59:59 EST
Sun 2020-03-08 03:00:00 EDT

Warning: The system is configured to read the RTC time in the local time zone.
This mode can not be fully supported. It will create various problems
with time zone changes and daylight saving time adjustments. The RTC
time is never updated, it relies on external facilities to maintain it.
If at all possible, use RTC in UTC by calling
'timedatectl set-local-rtc 0'.

重点是timezone 那一行, Asia/Shanghai 变成了EST了,明显不对, 我们需要用下面的命令修改一下:

timedatectl set-timezone "America/New_York"

重启以后,系统的各种时间才会恢复正常.

为什么要这么做?因为如果你不这么做,系统的时间和mysql 的时间会发生冲突,当你需要重启mysql服务的时候,就会出现:

InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

让你需要最起码5分钟以上才能完全重启mysql服务或者重启服务器。。。

下午遇到了问题,一台新装的ecs服务器需要最少5分钟才能重启。。。刚开始以为是smokeping 重启过慢,仔细debug了开机启动,发现是关机速度太慢。。。然后看log 发现是mysql shutdown 时间太长,InnoDB需要清空buffer。。。再继续debug才发现这是系统时间的问题。。。

浪费了整整一下午。。。

ERROR 2006 (HY000): MySQL server has gone away

当你要导入一个比较大的mysql数据库的时候,可能会产生这个错误:

mysql> source file.sql
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 2
Current database: *** NONE ***

ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 3
Current database: *** NONE ***

这是因为max_allowed_packet设置的太小了,再/etc/my.cnf中,将这个参数设置为64M或者根据实际情况设为更大的值就可以了

max_allowed_packet=64M

mysql文档:

https://dev.mysql.com/doc/refman/5.5/en/replication-features-max-allowed-packet.html

 

Mysql 快速备份和恢复

shell> mysqldump db1 > dump.sql
shell> mysqladmin create db2
shell> mysql db2 < dump.sql

Do not use –databases on the mysqldump command line because that causes USE db1 to be included in the dump file, which overrides the effect of naming db2 on the mysql command line.

有的时候root的密码不为空,这个时候我们就需要:

shell> mysqldump -uroot -p db1 > dump.sql

这个我们就可以把db1的表给倒出来了

MySql 的 ONLY_FULL_GROUP_BY

最近把mysql 升级到了5.7.18 然后就发现了一个很明显的变化, ONLY_FULL_GROUP_BY 这个mysql mode 被加了进来, 按照sql 99 的标准这是好事.

但是实际上, mysql 的这个版本只能说almost identical or very close to, 不能说完全相等

简单的说,就是

That is, MySQL 5.7.5m15 will by default reject only those GROUP 
BY-queries that include non-aggregated expressions in the SELECT-list that 
are not functionally dependent upon the GROUP BY-list.

This not only means that you cannot mess up your GROUP BY-queries anymore 
(as MySQL will now reject an improper GROUP BY query), 
it will also not require you to write non-sensical "dummy" aggregates
 over expressions that can only have one value per aggregated 
result row. Hurrah!

详细的解释请看这篇文章:

http://rpbouman.blogspot.nl/2014/09/mysql-575-group-by-respects-functional.html