Category: Nginx

在Nginx 的location中，if is evil.

因此多个if条件语句可以转化为多个map条件语句.

一般有两种方式，一个是map中直接map 两个变量，变量之间用:间隔；另外一个就是使用多个map，后一个map里面直接使用前面一个map里的变量，形成map chain. 下面详细说明:

第一种方式:

map "$http_x_target:$arg_target" $destination {
default upstream0;
~something upstream1;
~something2 upstream1;
~something3 upstream2;
}
...
server {
location / {
proxy_pass https://$destination;
}
}

第二种方式:

map $arg_target $arg_destination {
default upstream0;
something upstream1;
something2 upstream1;
something3 upstream2;
}
map $http_x_target $destination {
default $arg_destination;
something upstream1;
something2 upstream1;
something3 upstream2;
}
...
server {
location / {
proxy_pass https://$destination;
}
}

参考文档:

https://stackoverflow.com/questions/59671623/conditionally-map-values-in-nginx-config
https://gock.net/blog/2020/nginx-conditional-logging-responses/

一般来说，我们直接用nginx 的if 语句配合正则表达式就可以了，比如说

# case sensitive matching
if ($http_user_agent ~ (Antivirx|Arian)) {
return 403;
}

# case insensitive matching
if ($http_user_agent ~* (netcrawl|npbot|malicious)) {
return 403;
}

但是当我们需要禁止的user agent lists过长时，用if语句配合正则表达式就不是那么方便，而且性能上也会有影响，因此Nginx官方

多推荐使用map 来代替 if 语句

map $http_user_agent $badagent {
default 0;
~*malicious 1;
~*backdoor 1;
~*netcrawler 1;
~Antivirx 1;
~Arian 1;
~webbandit 1;
}

 if ($badagent) {
return 403;
}

Google weblight 说的很好听，可以让较慢网速的人访问你的网站，但是也有他自身的不足: 严重影响你的广告收入.

因此很多人都会想法去关闭Google weblight的访问.

Google weblight的网站给出方案，就是在header里面添加Cache-Control: no-transform. Google 看到这个header，就会展示原网页.

一个小知识, 按照RFC 7230， 3.2.2. Field Order里面所说，你可以在header里面添加多个Cache-Control，只要他们的field-value pair是不一样的，多个Cache-Control就是有效的.

比如说:

Cache-Control: no-cache, no-store, private
Cache-Control: no-cache

这是有效的，会被浏览器认为是:

Cache-Control: no-cache, no-store, private, no-cache

这个明白了以后, 在nginx中关闭Google Weblight 访问就很简单了. Google weblight使用的UA为

Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko; googleweblight) Chrome/38.0.1025.166 Mobile Safari/535.19

关键字符”googleweblight”, 因此我们只需要对含有googleweblight字符的UA的访问，添加header “Cache-Control: no-transform”就好了

下面上配置代码:

在Nginx http block 里面添加map 命令:

map $http_user_agent $cache_no_transform { 
"~*googleweblight" "no-transform"; 
}

然后在server block 或者 location block 里面添加：

add_header Cache-Control $cache_no_transform;

就可以了. 

一个小知识, Nginx 的add_header命令，如果value为空的话，默认是会忽略的. 因此

add_header Cache-Control '';

其实是会被nginx忽略的.

 

Nginx 支持websocket，可以用下面的example:

location / { 
       proxy_redirect off;
       proxy_pass http://127.0.0.1:26534; 
       proxy_http_version 1.1;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       proxy_set_header Host $http_host;
}

26534表示任意端口

分析nginx日志有各种各样的可视化工具，但是这样比较繁琐，需要安装和配置，大部分的时候我们只需要简单的分析，这里awk 完全可以满足我们的需求。

1. 统计日志中访问最多的10个ip

 方法一

awk '{a[$1]++}END{for(i in a)print a[i],i|"sort -k1 -nr|head -n10"}' access.log

方法二

awk '{print $1}' access.log |sort |uniq -c |sort -k1 -nr |head -n10

 

2. 统计日志中访问大于100次的IP

方法一

awk '{a[$1]++}END{for(i in a){if(a[i]>100)print i,a[i]}}' access.log

方法二

awk '{a[$1]++;if(a[$1]>100){b[$1]++}}END{for(i in b){print i,a[i]}}' access.log

3. 统计2019年12月24日一天内访问最多的10个IP

方法一

awk '$4>="[24/Dec/2019:00:00:01" && $4<="[24/Dec/2019:23:59:59" {a[$1]++}END{for(i in a)print a[i],i|"sort -k1 -nr|head -n10"}' access.log

方法二

sed -n '/\[24\/Dec\/2019:00:00:01/,/\[24\/Dec\/2019:23:59:59/p' access.log |sort |uniq -c |sort -k1 -nr |head -n10

4. 统计访问最多的前10个页面 ($request)

awk '{a[$7]++}END{for(i in a)print a[i],i|"sort -k1 -nr|head -n10"}' access.log

5. 统计蜘蛛抓取次数

grep 'Baiduspider' access.log |wc -l

统计蜘蛛抓取404的次数

grep 'Baiduspider' access.log |grep '404' | wc -l

 

 

基本说明见这里: https://www.iamhippo.com/2019-12/1167.html

1) update system and install building software

yum clean all
yum update -y

disable selinux

vi /etc/selinux/config
reboot
yum groupinstall -y 'Development Tools'

2) add nginx username and group, identical to the one nginx offical repo creates:

useradd --system --home /var/cache/nginx --shell /sbin/nologin --comment "nginx user" --user-group nginx

 check user and group created

3) download nginx dependencies source code

cd ~
wget https://ftp.pcre.org/pub/pcre/pcre-8.43.tar.gz && tar xzvf pcre-8.43.tar.gz
wget https://www.zlib.net/zlib-1.2.11.tar.gz && tar xzvf zlib-1.2.11.tar.gz
wget http://www.openssl.org/source/openssl-1.1.1d.tar.gz && tar xzvf openssl-1.1.1d.tar.gz

wget https://nginx.org/download/nginx-1.16.1.tar.gz && tar zxvf nginx-1.16.1.tar.gz
cd nginx-1.16.1
./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-pcre=../pcre-8.43 --with-pcre-jit --with-zlib=../zlib-1.2.11 --with-openssl=../openssl-1.1.1d --with-openssl-opt=no-nextprotoneg --with-debug

make
make install

# go to home

cd ~

# symlink /usr/lib64/nginx/modules to /etc/nginx/modules

ln -s /usr/lib64/nginx/modules /etc/nginx/modules
mkdir /var/cache/nginx -p
mkdir /etc/nginx/vhost -p

vi /usr/lib/systemd/system/nginx.service

[Unit]
Description=nginx - high performance web server
Documentation=http://nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -c /etc/nginx/nginx.conf
ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID

[Install]
WantedBy=multi-user.target

reload systemd daemon

systemctl daemon-reload

start service and auto boot

systemctl start nginx
systemctl enable nginx

check if nginx startup on reboot

systemctl is-enabled nginx.service

check if nginx is running

systemctl status nginx

对于在centos 7， debian 9/10 上自己编译的nginx来说，默认的nginx 配置有点弱，于是根据军哥的LNMP的配置，我做了一些修改. 以后凡是自己按照nginx官方repo 的configure编译的nginx，都可以使用如下的nginx.conf，需要在nginx.conf 所在的目录设置一个vhost目录, 所有的individual host 的配置，都放在vhost里面，方便管理. 

user  nginx nginx;

worker_processes auto;
worker_cpu_affinity auto;

error_log  /var/log/nginx/error.log  crit;

pid        /var/run/nginx.pid;

#Specifies the value for maximum file descriptors that can be opened by this process.
worker_rlimit_nofile 51200;

events
    {
        use epoll;
        worker_connections 51200;
        multi_accept off;
        accept_mutex off;
    }

http
    {
        include       mime.types;
        default_type  application/octet-stream;

        server_names_hash_bucket_size 128;
        client_header_buffer_size 32k;
        large_client_header_buffers 4 32k;
        client_max_body_size 50m;
		
        proxy_buffer_size 128k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;

        sendfile on;
        sendfile_max_chunk 512k;
        tcp_nopush on;

        keepalive_timeout 60;

        tcp_nodelay on;

        fastcgi_connect_timeout 300;
        fastcgi_send_timeout 300;
        fastcgi_read_timeout 300;
        fastcgi_buffer_size 64k;
        fastcgi_buffers 4 64k;
        fastcgi_busy_buffers_size 128k;
        fastcgi_temp_file_write_size 256k;

        gzip on;
        gzip_min_length  1k;
        gzip_buffers     4 16k;
        gzip_http_version 1.1;
        gzip_comp_level 2;
        gzip_types     text/plain application/javascript application/x-javascript text/javascript text/css application/xml application/xml+rss;
        gzip_vary on;
        gzip_proxied   expired no-cache no-store private auth;
        gzip_disable   "MSIE [1-6]\.";

        #limit_conn_zone $binary_remote_addr zone=perip:10m;
        ##If enable limit_conn_zone,add "limit_conn perip 10;" to server section.

        server_tokens off;
        access_log off;

server
    {
        listen 80 default_server;
        #listen [::]:80 default_server ipv6only=on;
        server_name _;
        index index.html index.htm;
        root  html;

        #error_page   404   /404.html;

        # Deny access to PHP files in specific directory
        #location ~ /(wp-content|uploads|wp-includes|images)/.*\.php$ { deny all; }

        #include enable-php.conf;

        location /nginx_status
        {
            stub_status on;
            access_log   off;
        }

        location ~ .*\.(gif|jpg|jpeg|png|bmp|swf)$
        {
            expires      30d;
        }

        location ~ .*\.(js|css)?$
        {
            expires      12h;
        }

        location ~ /.well-known {
            allow all;
        }

        location ~ /\.
        {
            deny all;
        }

#       access_log  /home/wwwlogs/access.log;
    }
include vhost/*.conf;
}